Nov 01 22:20:32.050781 ip-10-0-138-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 22:20:32.050796 ip-10-0-138-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 22:20:32.050804 ip-10-0-138-87 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 22:20:32.051146 ip-10-0-138-87 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 22:20:43.269993 ip-10-0-138-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 22:20:43.270011 ip-10-0-138-87 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f43fc64b03614ff0bb62e39f87d63435 -- Nov 01 22:22:41.637872 ip-10-0-138-87 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:22:41.984593 ip-10-0-138-87 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:41.984593 ip-10-0-138-87 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:22:41.984593 ip-10-0-138-87 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:41.984593 ip-10-0-138-87 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:22:41.989978 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989894 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:41.989978 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989978 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989982 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989986 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989988 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989991 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989994 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.989999 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990003 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990007 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990010 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990013 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990021 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990024 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990027 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990030 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990032 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990035 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990038 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990040 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:41.990051 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990043 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990046 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990048 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990051 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990053 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990056 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990059 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990062 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990065 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990067 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990070 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990073 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990075 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990078 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990080 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990083 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990085 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990088 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990090 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990094 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:41.990485 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990097 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990099 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990102 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990104 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990107 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990109 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990112 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990115 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990117 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990120 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990122 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990125 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990129 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990132 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990136 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990139 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990142 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990145 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990147 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:41.990975 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990150 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990153 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990155 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990158 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990160 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990163 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990165 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990168 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990170 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990173 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990176 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990178 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990181 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990184 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990187 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990189 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990192 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990194 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990197 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990200 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:41.991422 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990202 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990204 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990208 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990210 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990213 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990215 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990218 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990221 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990224 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990227 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990230 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990233 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990235 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990238 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990241 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990243 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990246 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990249 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990251 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990254 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:41.991909 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990256 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990259 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990261 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990264 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990266 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990269 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.990273 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991055 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991063 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991066 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991069 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991073 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991076 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991078 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991081 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991084 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991086 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991089 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991091 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991094 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:41.992442 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991097 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991100 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991102 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991105 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991107 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991110 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991113 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991115 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991118 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991121 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991123 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991126 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991129 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991131 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991134 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991137 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991139 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991142 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991144 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991148 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:41.992944 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991150 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991153 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991155 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991160 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991163 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991165 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991168 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991171 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991173 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991176 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991178 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991181 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991184 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991186 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991190 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991194 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991197 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991200 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:41.993419 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991202 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991205 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991207 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991210 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991212 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991215 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991218 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991220 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991223 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991228 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991232 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991235 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991238 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991241 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991243 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991246 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991249 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991251 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991254 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:41.993877 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991257 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991259 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991262 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991264 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991267 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991270 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991272 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991275 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991277 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991280 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991282 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991285 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991287 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991291 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991294 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991297 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991300 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991302 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991305 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:41.994322 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991310 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991312 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991315 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991317 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991320 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991322 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991325 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991328 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991330 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991333 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991335 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991338 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991340 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991343 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991345 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991853 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991859 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991865 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991868 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:41.994788 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991870 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991873 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991876 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991878 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991881 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991883 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991887 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991889 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991892 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991895 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991897 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991900 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991902 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991905 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991907 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991910 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991913 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991915 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991918 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991920 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:41.995254 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991924 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991926 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991929 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991931 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991934 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991936 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991940 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991944 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991947 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991950 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991955 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991957 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991960 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991962 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991965 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991967 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991970 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991973 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991975 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991978 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:41.995733 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991980 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991982 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991985 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991987 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991990 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991993 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991995 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.991998 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992000 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992003 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992005 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992008 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992010 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992013 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992016 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992019 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992022 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992024 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992027 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992029 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:41.996242 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992032 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992034 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992038 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992041 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992043 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992046 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992049 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992052 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992056 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992059 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992062 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992065 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992069 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992071 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992074 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992077 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992080 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992083 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992085 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:41.996732 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992087 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992090 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992093 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992095 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992098 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992100 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992102 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992111 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992114 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992117 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992119 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992122 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992124 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992127 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992129 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992134 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992137 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992140 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992142 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:41.997211 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992145 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992147 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992150 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.992152 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.992999 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993009 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993021 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993026 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993031 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993034 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993038 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993043 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993046 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993050 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993053 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993056 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993060 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993063 2578 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993066 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993069 2578 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993071 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993075 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993079 2578 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:22:41.997708 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993082 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993085 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993089 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993092 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993096 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993100 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993103 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993107 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993110 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993113 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993116 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993119 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993122 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993126 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993129 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993132 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993135 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993138 2578 flags.go:64] FLAG: --enable-server="true" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993141 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993145 2578 flags.go:64] FLAG: --event-burst="100" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993148 2578 flags.go:64] FLAG: --event-qps="50" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993151 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993154 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993157 2578 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:22:41.998244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993161 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993164 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993167 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993170 2578 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993173 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993176 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993179 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993182 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993185 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993188 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993191 2578 flags.go:64] FLAG: --feature-gates="" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993195 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993198 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993203 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993206 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993209 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993212 2578 flags.go:64] FLAG: --help="false" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993215 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993219 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993222 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993225 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993228 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993236 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:22:41.998886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993239 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993242 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993245 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993248 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993251 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993254 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993257 2578 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993260 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993263 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993266 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993269 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993272 2578 flags.go:64] FLAG: --lock-file="" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993275 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993278 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993281 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993287 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993289 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993293 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993296 2578 flags.go:64] FLAG: --logging-format="text" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993300 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993303 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993306 2578 flags.go:64] FLAG: --manifest-url="" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993311 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993316 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993320 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:22:41.999456 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993324 2578 flags.go:64] FLAG: --max-pods="110" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993327 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993330 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993333 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993336 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993339 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993342 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993345 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993352 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993355 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993358 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993361 2578 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993364 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993367 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993370 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993373 2578 flags.go:64] FLAG: --port="10250" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993377 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993379 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0dec6a2294b88ef75" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993383 2578 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993386 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993389 2578 flags.go:64] FLAG: --register-node="true" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993392 2578 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993396 2578 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993399 2578 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993402 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993406 2578 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:22:42.000138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993410 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993413 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993416 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993421 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993424 2578 flags.go:64] FLAG: --runonce="false" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993427 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993431 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993434 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993437 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993440 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993443 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993446 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993450 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993453 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993456 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993459 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993462 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993465 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993468 2578 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993471 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993477 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993479 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993482 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993486 2578 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993489 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:22:42.000825 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993492 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993495 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993498 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993500 2578 flags.go:64] FLAG: --v="2" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993505 2578 flags.go:64] FLAG: --version="false" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993509 2578 flags.go:64] FLAG: --vmodule="" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993513 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993516 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993671 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993676 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993682 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993685 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993688 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993691 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993693 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993697 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993699 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993702 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993705 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993708 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993711 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:42.001409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993714 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993717 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993719 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993722 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993725 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993728 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993732 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993736 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993739 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993741 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993744 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993747 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993750 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993752 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993755 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993757 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993760 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993762 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993765 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:42.001920 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993768 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993771 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993776 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993778 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993781 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993783 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993786 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993789 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993792 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993794 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993797 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993801 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993803 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993806 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993808 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993811 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993814 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993816 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993819 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:42.002395 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993821 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993824 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993826 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993829 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993832 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993834 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993837 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993839 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993842 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993844 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993847 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993849 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993852 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993854 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993858 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993861 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993864 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993867 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993869 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993872 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:42.002874 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993874 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993878 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993883 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993886 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993890 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993893 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993895 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993898 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993900 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993903 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993906 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993908 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993911 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993914 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993916 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993919 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993921 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993924 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993926 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:42.003346 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993929 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993931 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993934 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993936 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993939 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993941 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993944 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993946 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993951 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993954 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993957 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993959 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993962 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993964 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993967 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.993969 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:42.003860 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:41.993997 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994036 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994040 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994044 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994046 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994049 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994052 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994054 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994057 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994059 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994063 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994066 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994069 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994072 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994074 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994077 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994079 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994082 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994084 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994087 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:42.004264 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994089 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994091 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994094 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994097 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994099 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994103 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994106 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994109 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994111 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994114 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994116 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994119 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994121 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994126 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994130 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994132 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994136 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994138 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994141 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:42.004817 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994143 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994146 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994148 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994151 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994153 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994156 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994158 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994161 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994163 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994166 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994168 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994171 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994173 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994176 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994178 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994181 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994183 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994186 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994190 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:42.005498 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994193 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994196 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994198 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994201 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994203 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994206 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994209 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994212 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994215 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994220 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994222 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994225 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994227 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994230 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994232 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994236 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994238 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994241 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994243 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:42.006143 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994246 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994248 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994251 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994253 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994256 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994258 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994261 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994264 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994268 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994271 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994274 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994278 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994282 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994285 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994288 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994291 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994293 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994296 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994298 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:42.006756 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994302 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994304 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994307 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994310 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994314 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994316 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994319 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994322 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994324 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:41.994327 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.001396 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.001412 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.002069 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.002074 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002145 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002150 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002155 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002158 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002161 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:42.007409 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002163 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002166 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002168 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002171 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002174 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002176 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002179 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002182 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002185 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002187 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002190 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002192 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002196 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002198 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002201 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002203 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002206 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002208 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002211 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002213 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:42.007891 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002216 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002219 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002221 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002224 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002233 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002236 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002238 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002241 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002244 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002246 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002249 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002252 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002254 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002257 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002259 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002262 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002266 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002270 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002274 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:42.008628 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002276 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002279 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002281 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002284 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002287 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002289 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002292 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002294 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002297 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002299 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002302 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002304 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002307 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002309 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002312 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002315 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002317 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002320 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002328 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:42.009259 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002331 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002334 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002337 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002339 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002343 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002346 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002349 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002352 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002354 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002357 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002359 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002362 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002365 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002367 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002370 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002373 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002375 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002378 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002380 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:42.009765 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002383 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002385 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002388 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002390 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002393 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002395 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002398 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002401 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002403 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002407 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002409 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002413 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002415 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002425 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002428 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002431 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002433 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002436 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002438 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002441 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002443 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:42.010286 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002451 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002454 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002457 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.002485 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002686 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002691 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002694 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002698 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002700 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002703 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002706 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002708 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002711 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002714 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002716 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002719 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:42.010883 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002721 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002724 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002726 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002729 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002731 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002734 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002736 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002739 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002742 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002750 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002753 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002755 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002757 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002760 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002762 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002765 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002767 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002770 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002772 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002775 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:42.011380 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002777 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002779 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002782 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002784 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002787 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002789 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002792 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002794 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002797 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002800 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002802 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002805 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002807 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002810 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002812 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002815 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002817 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002820 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002822 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:42.012148 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002825 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002827 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002830 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002838 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002841 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002844 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002846 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002849 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002851 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002854 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002856 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002858 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002862 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002866 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002870 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002874 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002877 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002880 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002882 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:42.012607 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002885 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002888 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002890 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002893 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002895 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002898 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002901 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002903 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002905 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002908 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002910 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002913 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002916 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002918 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002921 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002924 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002926 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002935 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002938 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002940 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:42.013123 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002943 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002945 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002948 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002950 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002952 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002955 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002957 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002960 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002962 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002965 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002970 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002972 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002975 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002978 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:42.013856 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.002980 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.003004 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003043 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003047 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003050 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003052 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003055 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003058 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003060 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003063 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003065 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003068 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003071 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003073 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003076 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003084 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:22:42.014478 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003088 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003090 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003093 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003096 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003098 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003101 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003103 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003106 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003108 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003111 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003113 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003116 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003118 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003121 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003123 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003126 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003128 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003130 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003133 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003135 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:22:42.015209 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003138 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003140 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003143 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003148 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003151 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003153 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003156 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003160 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003162 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003165 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003168 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003170 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003179 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003182 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003184 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003187 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003189 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003192 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003195 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:22:42.015692 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003197 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003200 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003202 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003205 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003207 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003210 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003213 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003215 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003218 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003220 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003223 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003227 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003230 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003234 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003238 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003241 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003244 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003247 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003249 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:22:42.016177 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003252 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003256 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003259 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003261 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003264 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003266 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003269 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003278 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003281 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003284 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003286 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003289 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003292 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003295 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003300 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003303 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003305 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003308 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003310 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:22:42.016785 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003313 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003315 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003318 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003321 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003323 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003325 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003328 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003330 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003333 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003337 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003339 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003342 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003344 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:42.003348 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.003409 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.009092 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.009872 2578 server.go:1014] "Starting client certificate rotation" Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.009965 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:42.017298 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.010006 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:22:42.017848 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.011851 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:42.017848 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.015111 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:22:42.026356 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.026336 2578 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:22:42.029165 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.029151 2578 log.go:25] "Validated CRI v1 image API" Nov 01 22:22:42.030080 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.030052 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:22:42.033919 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.033898 2578 fs.go:135] Filesystem UUIDs: map[21a393e9-ecd7-498c-aa8c-f2de1e1e757b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d7de22f9-ddf5-4e77-9124-c3a7b309743c:/dev/nvme0n1p4] Nov 01 22:22:42.033985 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.033917 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 22:22:42.035559 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.035544 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:42.039810 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.039692 2578 manager.go:217] Machine: {Timestamp:2026-11-01 22:22:42.03822649 +0000 UTC m=+0.293028603 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3069363 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b7b716830ad78fe5215868d76f714 SystemUUID:ec2b7b71-6830-ad78-fe52-15868d76f714 BootID:f43fc64b-0361-4ff0-bb62-e39f87d63435 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:87:3c:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:87:3c:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:24:2b:10:95:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:22:42.040375 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.040365 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:22:42.040462 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.040447 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:22:42.042537 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.042513 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:22:42.042690 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.042538 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-87.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:22:42.042740 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.042692 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:22:42.042740 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.042700 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:22:42.042740 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.042738 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:42.042816 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.042756 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:22:42.043529 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.043520 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:22:42.047356 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.047345 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:22:42.047474 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.047466 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:22:42.047504 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.047495 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:22:42.047530 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.047508 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:22:42.047530 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.047517 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:22:42.047530 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.047526 2578 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:22:42.047618 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.047545 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:22:42.048275 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.048247 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:42.050755 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.050737 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:22:42.055412 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.055396 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tprs" Nov 01 22:22:42.056467 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.056454 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:22:42.057132 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057116 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:22:42.057170 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057164 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:22:42.057321 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057311 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:22:42.057349 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057327 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:22:42.057349 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057334 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:22:42.057349 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057340 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:22:42.057349 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057345 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:22:42.057349 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.057351 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:22:42.058193 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.058181 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:22:42.058227 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.058195 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:22:42.058227 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.058203 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:22:42.058227 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.058208 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:22:42.058227 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.058217 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:22:42.058227 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.058227 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:22:42.059207 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.059197 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:22:42.059244 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.059210 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:22:42.060911 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.060894 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2tprs" Nov 01 22:22:42.063087 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.063075 2578 server.go:1263] "Started kubelet" Nov 01 22:22:42.063327 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.063269 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:22:42.063407 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.063283 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:22:42.063469 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.063456 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:22:42.063910 ip-10-0-138-87 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:22:42.064873 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.064841 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:22:42.068070 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.068041 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 22:22:42.068177 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.068124 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-87.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 22:22:42.069024 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.069001 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-87.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 22:22:42.069557 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.069528 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 22:22:42.069701 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.069689 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:22:42.071401 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.071385 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:42.072281 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.072266 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:22:42.072376 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.072358 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.072831 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.072814 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:22:42.072913 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.072883 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:22:42.072913 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.072899 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:22:42.073018 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073006 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:22:42.073018 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073015 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:22:42.073118 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073103 2578 factory.go:55] Registering systemd factory Nov 01 22:22:42.073166 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073120 2578 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:22:42.073328 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073316 2578 factory.go:153] Registering CRI-O factory Nov 01 22:22:42.073391 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073332 2578 factory.go:223] Registration of the crio container factory successfully Nov 01 22:22:42.073391 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073385 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:22:42.073487 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073399 2578 factory.go:103] Registering Raw factory Nov 01 22:22:42.073487 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.073409 2578 manager.go:1199] Started watching for new ooms in manager Nov 01 22:22:42.074684 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.074672 2578 manager.go:319] Starting recovery of all containers Nov 01 22:22:42.075912 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.075893 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:42.080177 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.080150 2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-87.us-east-2.compute.internal\" not found" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.090737 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.089442 2578 manager.go:324] Recovery completed Nov 01 22:22:42.093426 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.093413 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:42.095231 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.095215 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:42.095311 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.095244 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:42.095311 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.095254 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:42.095796 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.095783 2578 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:22:42.095796 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.095792 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:22:42.095796 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.095794 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:42.095935 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.095808 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:22:42.096441 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.096428 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:42.096508 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.096454 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:42.096508 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.096464 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:42.098072 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.098060 2578 policy_none.go:50] "Start" Nov 01 22:22:42.098111 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.098075 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:22:42.098111 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.098085 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:22:42.243304 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.172775 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.243304 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.242415 2578 policy_none.go:44] "Start" Nov 01 22:22:42.273268 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.273077 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.279189 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.279158 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:22:42.281981 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.281966 2578 manager.go:341] "Starting Device Plugin manager" Nov 01 22:22:42.282057 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.282008 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:22:42.282057 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.282020 2578 server.go:87] "Starting device plugin registration server" Nov 01 22:22:42.282271 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.282259 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:22:42.282316 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.282273 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:22:42.282429 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.282405 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:22:42.282546 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.282530 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:22:42.282546 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.282543 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:22:42.283079 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.283049 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:22:42.283170 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.283100 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.286817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.286796 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:22:42.287980 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.287956 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:22:42.288059 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.287986 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:22:42.288059 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.288022 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:22:42.288157 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.288078 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 22:22:42.291548 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.291531 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:42.383049 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.383021 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:42.386337 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.386317 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:42.386417 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.386353 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:42.386417 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.386364 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:42.386417 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.386389 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.388499 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.388482 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal"] Nov 01 22:22:42.388563 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.388554 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:42.389832 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.389817 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:42.389884 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.389848 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:42.389884 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.389861 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:42.392128 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392116 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:42.392287 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392273 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.392337 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392304 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:42.392924 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392899 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:42.392924 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392924 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:42.392924 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392905 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:42.393084 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392935 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:42.393084 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392958 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:42.393084 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.392975 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:42.394870 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.394854 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.394924 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.394908 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-87.us-east-2.compute.internal\": node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.395108 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.395095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.395149 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.395118 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:22:42.396847 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.396832 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:22:42.396920 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.396859 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:22:42.396920 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.396869 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:22:42.410513 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.410495 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.425267 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.425253 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.428004 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.427986 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.475558 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.475528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.475680 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.475565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.475680 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.475606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bc77fd8b25c4cfddf6fa5b50fb587e4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"9bc77fd8b25c4cfddf6fa5b50fb587e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.511025 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.510954 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.576062 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.576032 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bc77fd8b25c4cfddf6fa5b50fb587e4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"9bc77fd8b25c4cfddf6fa5b50fb587e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.576188 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.576091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.576188 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.576127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.576188 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.576135 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bc77fd8b25c4cfddf6fa5b50fb587e4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"9bc77fd8b25c4cfddf6fa5b50fb587e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.576188 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.576179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.576341 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.576205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.611186 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.611158 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.711711 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.711631 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.729823 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.729799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.732971 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:42.732951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:42.812417 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.812343 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:42.912874 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:42.912827 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:43.010311 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.010267 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 22:22:43.010844 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.010479 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:22:43.010844 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.010492 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 22:22:43.013505 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:43.013487 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:43.065980 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.065907 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:42 +0000 UTC" deadline="2028-04-11 00:50:12.967766911 +0000 UTC" Nov 01 22:22:43.065980 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.065937 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12626h27m29.901832214s" Nov 01 22:22:43.113957 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:43.113920 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:43.214806 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:43.214768 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:43.315654 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:43.315628 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:43.416538 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:43.416467 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-87.us-east-2.compute.internal\" not found" Nov 01 22:22:43.435797 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:43.435741 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc77fd8b25c4cfddf6fa5b50fb587e4.slice/crio-0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7 WatchSource:0}: Error finding container 0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7: Status 404 returned error can't find the container with id 0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7 Nov 01 22:22:43.440693 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.440671 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:43.441237 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:43.441213 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda61bce4694bad4d0a8e6759feac37d.slice/crio-4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3 WatchSource:0}: Error finding container 4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3: Status 404 returned error can't find the container with id 4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3 Nov 01 22:22:43.442125 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.442110 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:22:43.444113 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.444097 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:43.473033 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.473004 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:43.484520 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.484492 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:43.485369 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.485354 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:22:43.494655 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:43.494630 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:22:44.048714 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.048685 2578 apiserver.go:52] "Watching apiserver" Nov 01 22:22:44.057019 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.056992 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:22:44.057473 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.057441 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk","openshift-network-diagnostics/network-check-target-mpjqm","kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal","openshift-dns/node-resolver-n8wtc","openshift-multus/multus-additional-cni-plugins-bbtw6","openshift-multus/network-metrics-daemon-x44vh","openshift-network-operator/iptables-alerter-bwft6","kube-system/global-pull-secret-syncer-wdkm9","openshift-cluster-node-tuning-operator/tuned-wrllx","openshift-image-registry/node-ca-tnr7k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal","openshift-multus/multus-gxf42","openshift-ovn-kubernetes/ovnkube-node-xglkp","kube-system/konnectivity-agent-2rvph"] Nov 01 22:22:44.060416 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.060392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.063129 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.063105 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.064453 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.064434 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:44.064552 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.064545 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sq9j\"" Nov 01 22:22:44.064635 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.064619 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:44.064748 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.064729 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:22:44.065765 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.065535 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.066696 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.066284 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:22:44.066696 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.066403 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:22:44.066696 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.066587 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:22:44.066899 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.066749 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:22:44.066899 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.066818 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:22:44.067065 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.067046 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vwc2\"" Nov 01 22:22:44.068076 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.068055 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:22:44.068588 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.068549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.068721 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.068699 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9gzz5\"" Nov 01 22:22:44.069492 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.069475 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:22:44.069575 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.069508 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:22:44.071932 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.071912 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z74s5\"" Nov 01 22:22:44.072028 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.071996 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:22:44.072086 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.072050 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 22:22:44.072086 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.072068 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:22:44.073587 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.073569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.075925 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.075907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:44.076026 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.075991 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:44.079082 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.079046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.082074 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.082050 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.085338 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.085444 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbpk\" (UniqueName: \"kubernetes.io/projected/e91e58ba-dbc6-49da-b601-22edc7b6525e-kube-api-access-fdbpk\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.085505 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7025fde7-1830-4d2e-9780-a4c0a8195d0f-agent-certs\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.085505 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7025fde7-1830-4d2e-9780-a4c0a8195d0f-konnectivity-ca\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.085602 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-system-cni-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.085602 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.085602 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-env-overrides\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.085806 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-kubelet\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.085806 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085629 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.085806 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-etc-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.085806 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086047 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-netd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086047 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovn-node-metrics-cert\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086047 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.085945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cnibin\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.086047 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.086306 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-slash\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086306 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-bin\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086306 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/948069e6-a5b8-407d-a12b-198a76a2e783-hosts-file\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.086457 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-script-lib\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086457 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55hgz\" (UniqueName: \"kubernetes.io/projected/948069e6-a5b8-407d-a12b-198a76a2e783-kube-api-access-55hgz\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.086457 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-log-socket\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086616 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086616 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086570 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 22:22:44.086740 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-config\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086740 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7eeac783-1128-4402-8404-5962939c1b63-iptables-alerter-script\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.086848 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eeac783-1128-4402-8404-5962939c1b63-host-slash\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.086848 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-netns\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086949 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-systemd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.086949 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-ovn\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.087051 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:44.087051 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.086981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82s88\" (UniqueName: \"kubernetes.io/projected/7eeac783-1128-4402-8404-5962939c1b63-kube-api-access-82s88\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.087051 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.087015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-systemd-units\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.087051 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.087041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-var-lib-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.087051 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.087043 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:44.087314 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.087072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-node-log\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.087314 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.087197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948069e6-a5b8-407d-a12b-198a76a2e783-tmp-dir\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.087314 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.087258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.087314 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.087307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5z4\" (UniqueName: \"kubernetes.io/projected/a4ce9767-19f2-42a8-afe1-ad361726b8b9-kube-api-access-xk5z4\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.091364 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.091341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.093682 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.093660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.093762 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.093728 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:44.096850 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.096411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.102177 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.102156 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xl64d" Nov 01 22:22:44.111693 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.111672 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xl64d" Nov 01 22:22:44.173383 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173347 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:22:44.173383 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173378 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:22:44.173591 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173406 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:22:44.173591 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173485 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:22:44.173591 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173493 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:22:44.173591 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:22:44.173817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173636 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:22:44.173817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173656 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:22:44.173817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173721 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:22:44.173817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173786 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:22:44.174025 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.173928 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jvq4g\"" Nov 01 22:22:44.174087 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.174063 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nz8z8\"" Nov 01 22:22:44.174415 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.174398 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:22:44.174697 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.174675 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:22:44.175014 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.174992 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdns4\"" Nov 01 22:22:44.175138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.175053 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:22:44.175138 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.175128 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:22:44.175258 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.175140 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:22:44.175258 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.175185 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-th659\"" Nov 01 22:22:44.175360 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.175322 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzblw\"" Nov 01 22:22:44.187466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-bin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.187614 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-kubelet\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.187614 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887gs\" (UniqueName: \"kubernetes.io/projected/aa89357c-1284-4f7a-b319-fcdb18edb560-kube-api-access-887gs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.187614 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-etc-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.187785 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.187785 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187689 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-etc-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.187785 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-netd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.187785 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187762 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-netd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.187988 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovn-node-metrics-cert\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.187988 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-conf\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.187988 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187852 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-tmp\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.187988 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-kubelet-config\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.187988 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.187988 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.187988 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.187967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-slash\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/948069e6-a5b8-407d-a12b-198a76a2e783-hosts-file\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-systemd\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-slash\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/948069e6-a5b8-407d-a12b-198a76a2e783-hosts-file\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-var-lib-kubelet\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f130f8c5-0161-4ef2-8634-cc150d09ddf0-host\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188159 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:44.188280 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbtt\" (UniqueName: \"kubernetes.io/projected/47eb4322-3268-4e50-80e0-35f2dd802b0b-kube-api-access-qrbtt\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:44.188600 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-etc-selinux\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.188600 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-script-lib\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.188600 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:44.188600 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-socket-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.188600 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-cnibin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.188600 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-socket-dir-parent\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.188600 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188573 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-multus-certs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.188939 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.188939 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7eeac783-1128-4402-8404-5962939c1b63-iptables-alerter-script\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.188939 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-ovn\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.188939 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188793 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-registration-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.188939 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-ovn\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.188939 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-multus\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.189232 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188977 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-script-lib\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.189232 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.188995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82s88\" (UniqueName: \"kubernetes.io/projected/7eeac783-1128-4402-8404-5962939c1b63-kube-api-access-82s88\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.189232 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-var-lib-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.189232 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-var-lib-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.189232 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysconfig\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.189232 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5z4\" (UniqueName: \"kubernetes.io/projected/a4ce9767-19f2-42a8-afe1-ad361726b8b9-kube-api-access-xk5z4\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.189232 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189218 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7eeac783-1128-4402-8404-5962939c1b63-iptables-alerter-script\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-modprobe-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f130f8c5-0161-4ef2-8634-cc150d09ddf0-serviceca\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-system-cni-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-run\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-host\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189445 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-system-cni-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5b8\" (UniqueName: \"kubernetes.io/projected/f130f8c5-0161-4ef2-8634-cc150d09ddf0-kube-api-access-cl5b8\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.189578 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-netns\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-hostroot\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-sys\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zkk\" (UniqueName: \"kubernetes.io/projected/548c3a65-3217-4609-b266-b7eacca269a3-kube-api-access-56zkk\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-conf-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cnibin\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-bin\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-cni-binary-copy\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189842 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cnibin\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189854 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-etc-kubernetes\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hgz\" (UniqueName: \"kubernetes.io/projected/948069e6-a5b8-407d-a12b-198a76a2e783-kube-api-access-55hgz\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-bin\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-lib-modules\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-k8s-cni-cncf-io\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.189986 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.189982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-node-log\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948069e6-a5b8-407d-a12b-198a76a2e783-tmp-dir\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-kubernetes\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190065 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-log-socket\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190029 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-node-log\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-config\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190170 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-log-socket\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eeac783-1128-4402-8404-5962939c1b63-host-slash\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-netns\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-systemd-units\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-netns\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-systemd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-device-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190434 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-systemd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.190773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-sys-fs\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpmt\" (UniqueName: \"kubernetes.io/projected/0f3425f3-64ee-4134-b096-10e48c0c5faa-kube-api-access-lcpmt\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-system-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-daemon-config\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-etc-tuned\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.190597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbpk\" (UniqueName: \"kubernetes.io/projected/e91e58ba-dbc6-49da-b601-22edc7b6525e-kube-api-access-fdbpk\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7025fde7-1830-4d2e-9780-a4c0a8195d0f-agent-certs\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7025fde7-1830-4d2e-9780-a4c0a8195d0f-konnectivity-ca\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948069e6-a5b8-407d-a12b-198a76a2e783-tmp-dir\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.191323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-env-overrides\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.191717 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-kubelet\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.191717 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.192324 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.192324 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191975 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.192324 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.191985 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.192324 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.192066 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-kubelet\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.192324 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.192281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovn-node-metrics-cert\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.192655 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.192374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-dbus\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.192655 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.192613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7025fde7-1830-4d2e-9780-a4c0a8195d0f-konnectivity-ca\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.192655 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.192622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-systemd-units\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.192817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.192681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eeac783-1128-4402-8404-5962939c1b63-host-slash\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.192817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.192774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.193155 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.193129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-env-overrides\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.193155 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.193143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-config\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.197106 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.197085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7025fde7-1830-4d2e-9780-a4c0a8195d0f-agent-certs\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.199131 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.199104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82s88\" (UniqueName: \"kubernetes.io/projected/7eeac783-1128-4402-8404-5962939c1b63-kube-api-access-82s88\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.200748 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.200723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbpk\" (UniqueName: \"kubernetes.io/projected/e91e58ba-dbc6-49da-b601-22edc7b6525e-kube-api-access-fdbpk\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.201446 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.201418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hgz\" (UniqueName: \"kubernetes.io/projected/948069e6-a5b8-407d-a12b-198a76a2e783-kube-api-access-55hgz\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.203127 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.203100 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5z4\" (UniqueName: \"kubernetes.io/projected/a4ce9767-19f2-42a8-afe1-ad361726b8b9-kube-api-access-xk5z4\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.292721 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zkk\" (UniqueName: \"kubernetes.io/projected/548c3a65-3217-4609-b266-b7eacca269a3-kube-api-access-56zkk\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.292721 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-conf-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-cni-binary-copy\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-etc-kubernetes\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-lib-modules\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-k8s-cni-cncf-io\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292833 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-conf-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-kubernetes\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-etc-kubernetes\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-device-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.292933 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292934 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-k8s-cni-cncf-io\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-device-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-sys-fs\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.292996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpmt\" (UniqueName: \"kubernetes.io/projected/0f3425f3-64ee-4134-b096-10e48c0c5faa-kube-api-access-lcpmt\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-kubernetes\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-system-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-daemon-config\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-lib-modules\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293065 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-sys-fs\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-etc-tuned\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-dbus\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-bin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-kubelet\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887gs\" (UniqueName: \"kubernetes.io/projected/aa89357c-1284-4f7a-b319-fcdb18edb560-kube-api-access-887gs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-conf\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.293299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-tmp\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-kubelet-config\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-systemd\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-var-lib-kubelet\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f130f8c5-0161-4ef2-8634-cc150d09ddf0-host\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbtt\" (UniqueName: \"kubernetes.io/projected/47eb4322-3268-4e50-80e0-35f2dd802b0b-kube-api-access-qrbtt\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293394 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-cni-binary-copy\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-etc-selinux\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-dbus\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293474 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-systemd\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-socket-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.293516 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-cnibin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-var-lib-kubelet\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.294182 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-socket-dir-parent\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.293568 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret podName:7caa0db8-4f8f-4848-a2a6-845f20c9a888 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:44.793552675 +0000 UTC m=+3.048354809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret") pod "global-pull-secret-syncer-wdkm9" (UID: "7caa0db8-4f8f-4848-a2a6-845f20c9a888") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-daemon-config\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.293637 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-socket-dir-parent\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-system-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-bin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-conf\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-kubelet-config\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-socket-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293864 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-cnibin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293908 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-kubelet\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293974 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-etc-selinux\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.293982 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs podName:47eb4322-3268-4e50-80e0-35f2dd802b0b nodeName:}" failed. No retries permitted until 2026-11-01 22:22:44.793965221 +0000 UTC m=+3.048767326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs") pod "network-metrics-daemon-x44vh" (UID: "47eb4322-3268-4e50-80e0-35f2dd802b0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.293978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f130f8c5-0161-4ef2-8634-cc150d09ddf0-host\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-multus-certs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.294895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294056 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-registration-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-multus\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysconfig\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-modprobe-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294196 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-registration-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f130f8c5-0161-4ef2-8634-cc150d09ddf0-serviceca\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-run\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294241 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysconfig\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-host\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-multus-certs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294324 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-modprobe-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-run\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-host\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5b8\" (UniqueName: \"kubernetes.io/projected/f130f8c5-0161-4ef2-8634-cc150d09ddf0-kube-api-access-cl5b8\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-multus\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-netns\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.295466 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-hostroot\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.296203 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-sys\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.296203 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-hostroot\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.296203 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-netns\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.296203 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294585 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-sys\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.296203 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.294683 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f130f8c5-0161-4ef2-8634-cc150d09ddf0-serviceca\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.296441 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.296226 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-tmp\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.296441 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.296346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-etc-tuned\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.301268 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.301214 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:44.301268 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.301239 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:44.301268 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.301253 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwz2q for pod openshift-network-diagnostics/network-check-target-mpjqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:44.301466 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.301308 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q podName:3d4713d5-cc55-4d7e-86f8-f1e17f3b3858 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:44.801291707 +0000 UTC m=+3.056093811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wwz2q" (UniqueName: "kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q") pod "network-check-target-mpjqm" (UID: "3d4713d5-cc55-4d7e-86f8-f1e17f3b3858") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:44.303792 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.303763 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbtt\" (UniqueName: \"kubernetes.io/projected/47eb4322-3268-4e50-80e0-35f2dd802b0b-kube-api-access-qrbtt\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:44.304266 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.304166 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"bda61bce4694bad4d0a8e6759feac37d","Type":"ContainerStarted","Data":"4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3"} Nov 01 22:22:44.304469 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.304447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887gs\" (UniqueName: \"kubernetes.io/projected/aa89357c-1284-4f7a-b319-fcdb18edb560-kube-api-access-887gs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.306389 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.306358 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"9bc77fd8b25c4cfddf6fa5b50fb587e4","Type":"ContainerStarted","Data":"0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7"} Nov 01 22:22:44.308094 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.308073 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:44.308783 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.308739 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zkk\" (UniqueName: \"kubernetes.io/projected/548c3a65-3217-4609-b266-b7eacca269a3-kube-api-access-56zkk\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.310381 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.310359 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpmt\" (UniqueName: \"kubernetes.io/projected/0f3425f3-64ee-4134-b096-10e48c0c5faa-kube-api-access-lcpmt\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.310497 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.310476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5b8\" (UniqueName: \"kubernetes.io/projected/f130f8c5-0161-4ef2-8634-cc150d09ddf0-kube-api-access-cl5b8\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.333889 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.333859 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:22:44.380237 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.379981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:22:44.389885 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.389858 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:44.392332 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.392296 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeac783_1128_4402_8404_5962939c1b63.slice/crio-90332090f685da3f426e853bbc93d6a110a17f8889027bf81fa4d9995dc28dfa WatchSource:0}: Error finding container 90332090f685da3f426e853bbc93d6a110a17f8889027bf81fa4d9995dc28dfa: Status 404 returned error can't find the container with id 90332090f685da3f426e853bbc93d6a110a17f8889027bf81fa4d9995dc28dfa Nov 01 22:22:44.400661 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.400618 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7025fde7_1830_4d2e_9780_a4c0a8195d0f.slice/crio-7dbb553234b4354d81ac73f479623608c1e4605a15ebca764e4daa16798dae23 WatchSource:0}: Error finding container 7dbb553234b4354d81ac73f479623608c1e4605a15ebca764e4daa16798dae23: Status 404 returned error can't find the container with id 7dbb553234b4354d81ac73f479623608c1e4605a15ebca764e4daa16798dae23 Nov 01 22:22:44.410640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.410619 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:22:44.413810 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.413785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:22:44.422392 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.422364 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948069e6_a5b8_407d_a12b_198a76a2e783.slice/crio-cb86560c6b8a26ab17c9105a237a46b708eaa0e2d4c4de0e35dcfe1babf79868 WatchSource:0}: Error finding container cb86560c6b8a26ab17c9105a237a46b708eaa0e2d4c4de0e35dcfe1babf79868: Status 404 returned error can't find the container with id cb86560c6b8a26ab17c9105a237a46b708eaa0e2d4c4de0e35dcfe1babf79868 Nov 01 22:22:44.426088 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.426065 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91e58ba_dbc6_49da_b601_22edc7b6525e.slice/crio-cf55523618275778e7c088a4918a47a403f3dd406d6361c5f4bf37e538dd816b WatchSource:0}: Error finding container cf55523618275778e7c088a4918a47a403f3dd406d6361c5f4bf37e538dd816b: Status 404 returned error can't find the container with id cf55523618275778e7c088a4918a47a403f3dd406d6361c5f4bf37e538dd816b Nov 01 22:22:44.427309 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.427286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:22:44.432668 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.432646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:22:44.438416 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.438386 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548c3a65_3217_4609_b266_b7eacca269a3.slice/crio-2ea5fce4d96fd992f7f8a8dc5eb961c8531fc2672423dcb3a865aba69cf53913 WatchSource:0}: Error finding container 2ea5fce4d96fd992f7f8a8dc5eb961c8531fc2672423dcb3a865aba69cf53913: Status 404 returned error can't find the container with id 2ea5fce4d96fd992f7f8a8dc5eb961c8531fc2672423dcb3a865aba69cf53913 Nov 01 22:22:44.439566 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.439545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:22:44.441965 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.441835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:22:44.444867 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.444757 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ce9767_19f2_42a8_afe1_ad361726b8b9.slice/crio-7370cc4b591de15db84fba0d4bf74d6fe09ed0143ce859685086c3405df4714b WatchSource:0}: Error finding container 7370cc4b591de15db84fba0d4bf74d6fe09ed0143ce859685086c3405df4714b: Status 404 returned error can't find the container with id 7370cc4b591de15db84fba0d4bf74d6fe09ed0143ce859685086c3405df4714b Nov 01 22:22:44.446365 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.446345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gxf42" Nov 01 22:22:44.452219 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.452187 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3425f3_64ee_4134_b096_10e48c0c5faa.slice/crio-fa635b1ed172ae10c482eee2fe146282932248b864e9416a24975283e3c54ea3 WatchSource:0}: Error finding container fa635b1ed172ae10c482eee2fe146282932248b864e9416a24975283e3c54ea3: Status 404 returned error can't find the container with id fa635b1ed172ae10c482eee2fe146282932248b864e9416a24975283e3c54ea3 Nov 01 22:22:44.460436 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.458439 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf130f8c5_0161_4ef2_8634_cc150d09ddf0.slice/crio-f73965ed56ae4f813336ce927838ac356df9e7144711904f5a38304c1842f781 WatchSource:0}: Error finding container f73965ed56ae4f813336ce927838ac356df9e7144711904f5a38304c1842f781: Status 404 returned error can't find the container with id f73965ed56ae4f813336ce927838ac356df9e7144711904f5a38304c1842f781 Nov 01 22:22:44.460436 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:22:44.460282 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa89357c_1284_4f7a_b319_fcdb18edb560.slice/crio-a8332fdaf88ac16b69fab9b38a0b8ced211975b1ae3e93d3142a5ddac2f1de62 WatchSource:0}: Error finding container a8332fdaf88ac16b69fab9b38a0b8ced211975b1ae3e93d3142a5ddac2f1de62: Status 404 returned error can't find the container with id a8332fdaf88ac16b69fab9b38a0b8ced211975b1ae3e93d3142a5ddac2f1de62 Nov 01 22:22:44.798951 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.798848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:44.798951 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.798909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:44.799157 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.799081 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:44.799157 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.799146 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs podName:47eb4322-3268-4e50-80e0-35f2dd802b0b nodeName:}" failed. No retries permitted until 2026-11-01 22:22:45.799128582 +0000 UTC m=+4.053930700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs") pod "network-metrics-daemon-x44vh" (UID: "47eb4322-3268-4e50-80e0-35f2dd802b0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:44.799641 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.799613 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:44.799776 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.799685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret podName:7caa0db8-4f8f-4848-a2a6-845f20c9a888 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:45.799651596 +0000 UTC m=+4.054453697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret") pod "global-pull-secret-syncer-wdkm9" (UID: "7caa0db8-4f8f-4848-a2a6-845f20c9a888") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:44.899918 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:44.899885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:44.900116 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.900046 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:44.900116 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.900066 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:44.900116 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.900078 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwz2q for pod openshift-network-diagnostics/network-check-target-mpjqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:44.900284 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:44.900139 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q podName:3d4713d5-cc55-4d7e-86f8-f1e17f3b3858 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:45.900120009 +0000 UTC m=+4.154922124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwz2q" (UniqueName: "kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q") pod "network-check-target-mpjqm" (UID: "3d4713d5-cc55-4d7e-86f8-f1e17f3b3858") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:45.115126 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.115014 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:44 +0000 UTC" deadline="2028-08-04 15:19:44.737857182 +0000 UTC" Nov 01 22:22:45.115126 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.115059 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15400h56m59.622803501s" Nov 01 22:22:45.288964 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.288931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:45.289144 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.289085 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:45.289537 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.289514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:45.289644 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.289617 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:45.289736 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.289704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:45.289801 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.289779 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:45.320197 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.320154 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rvph" event={"ID":"7025fde7-1830-4d2e-9780-a4c0a8195d0f","Type":"ContainerStarted","Data":"7dbb553234b4354d81ac73f479623608c1e4605a15ebca764e4daa16798dae23"} Nov 01 22:22:45.345011 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.344892 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwft6" event={"ID":"7eeac783-1128-4402-8404-5962939c1b63","Type":"ContainerStarted","Data":"90332090f685da3f426e853bbc93d6a110a17f8889027bf81fa4d9995dc28dfa"} Nov 01 22:22:45.353841 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.353710 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" event={"ID":"0f3425f3-64ee-4134-b096-10e48c0c5faa","Type":"ContainerStarted","Data":"fa635b1ed172ae10c482eee2fe146282932248b864e9416a24975283e3c54ea3"} Nov 01 22:22:45.361066 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.360935 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"cf55523618275778e7c088a4918a47a403f3dd406d6361c5f4bf37e538dd816b"} Nov 01 22:22:45.363686 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.363597 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n8wtc" event={"ID":"948069e6-a5b8-407d-a12b-198a76a2e783","Type":"ContainerStarted","Data":"cb86560c6b8a26ab17c9105a237a46b708eaa0e2d4c4de0e35dcfe1babf79868"} Nov 01 22:22:45.365366 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.365263 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxf42" event={"ID":"aa89357c-1284-4f7a-b319-fcdb18edb560","Type":"ContainerStarted","Data":"a8332fdaf88ac16b69fab9b38a0b8ced211975b1ae3e93d3142a5ddac2f1de62"} Nov 01 22:22:45.366786 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.366701 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnr7k" event={"ID":"f130f8c5-0161-4ef2-8634-cc150d09ddf0","Type":"ContainerStarted","Data":"f73965ed56ae4f813336ce927838ac356df9e7144711904f5a38304c1842f781"} Nov 01 22:22:45.370628 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.369738 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerStarted","Data":"7370cc4b591de15db84fba0d4bf74d6fe09ed0143ce859685086c3405df4714b"} Nov 01 22:22:45.379829 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.379798 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wrllx" event={"ID":"548c3a65-3217-4609-b266-b7eacca269a3","Type":"ContainerStarted","Data":"2ea5fce4d96fd992f7f8a8dc5eb961c8531fc2672423dcb3a865aba69cf53913"} Nov 01 22:22:45.808909 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.808413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:45.809089 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.808938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:45.809156 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.808851 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:45.809219 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.809177 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret podName:7caa0db8-4f8f-4848-a2a6-845f20c9a888 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:47.809158654 +0000 UTC m=+6.063960773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret") pod "global-pull-secret-syncer-wdkm9" (UID: "7caa0db8-4f8f-4848-a2a6-845f20c9a888") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:45.809524 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.809114 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:45.809682 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.809668 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs podName:47eb4322-3268-4e50-80e0-35f2dd802b0b nodeName:}" failed. No retries permitted until 2026-11-01 22:22:47.809617278 +0000 UTC m=+6.064419397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs") pod "network-metrics-daemon-x44vh" (UID: "47eb4322-3268-4e50-80e0-35f2dd802b0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:45.910109 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:45.909955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:45.910269 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.910148 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:45.910269 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.910171 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:45.910269 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.910184 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwz2q for pod openshift-network-diagnostics/network-check-target-mpjqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:45.910269 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:45.910243 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q podName:3d4713d5-cc55-4d7e-86f8-f1e17f3b3858 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:47.910224745 +0000 UTC m=+6.165026879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwz2q" (UniqueName: "kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q") pod "network-check-target-mpjqm" (UID: "3d4713d5-cc55-4d7e-86f8-f1e17f3b3858") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:46.115919 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:46.115855 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:44 +0000 UTC" deadline="2028-06-06 13:10:43.357792324 +0000 UTC" Nov 01 22:22:46.115919 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:46.115916 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13982h47m57.241880181s" Nov 01 22:22:46.386859 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:46.385974 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"9bc77fd8b25c4cfddf6fa5b50fb587e4","Type":"ContainerStarted","Data":"e543a73b6a39072e910b53060d36acb8d4f70cc851a2a004fb97032535af903d"} Nov 01 22:22:47.289487 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.288665 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:47.289487 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.288815 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:47.289487 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.289212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:47.289487 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.289305 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:47.289487 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.289362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:47.289487 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.289436 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:47.394021 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.392672 2578 generic.go:356] "Generic (PLEG): container finished" podID="bda61bce4694bad4d0a8e6759feac37d" containerID="bc19ee39eaf2d65b3712dbd0f6e8285ed9cb9a2a2ec2297b6889f61075b30d67" exitCode=0 Nov 01 22:22:47.394021 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.393748 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"bda61bce4694bad4d0a8e6759feac37d","Type":"ContainerDied","Data":"bc19ee39eaf2d65b3712dbd0f6e8285ed9cb9a2a2ec2297b6889f61075b30d67"} Nov 01 22:22:47.410870 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.410792 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" podStartSLOduration=3.410770254 podStartE2EDuration="3.410770254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:46.404611712 +0000 UTC m=+4.659413837" watchObservedRunningTime="2026-11-01 22:22:47.410770254 +0000 UTC m=+5.665572380" Nov 01 22:22:47.828476 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.828414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:47.828476 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.828480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:47.828734 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.828668 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:47.828734 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.828735 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs podName:47eb4322-3268-4e50-80e0-35f2dd802b0b nodeName:}" failed. No retries permitted until 2026-11-01 22:22:51.82871414 +0000 UTC m=+10.083516263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs") pod "network-metrics-daemon-x44vh" (UID: "47eb4322-3268-4e50-80e0-35f2dd802b0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:47.829863 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.829151 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:47.829863 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.829206 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret podName:7caa0db8-4f8f-4848-a2a6-845f20c9a888 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:51.829190389 +0000 UTC m=+10.083992505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret") pod "global-pull-secret-syncer-wdkm9" (UID: "7caa0db8-4f8f-4848-a2a6-845f20c9a888") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:47.929438 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:47.929370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:47.929622 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.929547 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:47.929622 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.929567 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:47.929622 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.929580 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwz2q for pod openshift-network-diagnostics/network-check-target-mpjqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:47.929791 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:47.929661 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q podName:3d4713d5-cc55-4d7e-86f8-f1e17f3b3858 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:51.929641083 +0000 UTC m=+10.184443200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwz2q" (UniqueName: "kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q") pod "network-check-target-mpjqm" (UID: "3d4713d5-cc55-4d7e-86f8-f1e17f3b3858") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:48.408217 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:48.408138 2578 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="cf314368adfae09b3102f588be31f648f6316bda6d9a3051783e4222d8a1d921" exitCode=0 Nov 01 22:22:48.408703 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:48.408229 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerDied","Data":"cf314368adfae09b3102f588be31f648f6316bda6d9a3051783e4222d8a1d921"} Nov 01 22:22:48.418907 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:48.418744 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"bda61bce4694bad4d0a8e6759feac37d","Type":"ContainerStarted","Data":"4efb31777281f0fd89553d0cd588edb64286cdce0d10382dc2e94cd2f5167423"} Nov 01 22:22:49.289428 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:49.288525 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:49.289428 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:49.288689 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:49.289428 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:49.289133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:49.289428 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:49.289227 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:49.289428 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:49.289289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:49.289428 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:49.289375 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:49.427714 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:49.427627 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rvph" event={"ID":"7025fde7-1830-4d2e-9780-a4c0a8195d0f","Type":"ContainerStarted","Data":"e34788b486a3ff7133611739e5c6b657911a5858cdb279460a9a334433c48868"} Nov 01 22:22:49.443720 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:49.443177 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" podStartSLOduration=5.443157982 podStartE2EDuration="5.443157982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:48.489685771 +0000 UTC m=+6.744487896" watchObservedRunningTime="2026-11-01 22:22:49.443157982 +0000 UTC m=+7.697960110" Nov 01 22:22:49.443720 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:49.443613 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2rvph" podStartSLOduration=2.188304488 podStartE2EDuration="6.443601194s" totalImagesPullingTime="4.255296706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:49.442814526 +0000 UTC m=+7.697616652" watchObservedRunningTime="2026-11-01 22:22:49.443601194 +0000 UTC m=+7.698403323" Nov 01 22:22:50.435878 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:50.435829 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnr7k" event={"ID":"f130f8c5-0161-4ef2-8634-cc150d09ddf0","Type":"ContainerStarted","Data":"8daa1db6764b7c7ba248d8f82b8813a93d5ce15f759e38ac3588f6d1c8ae6b08"} Nov 01 22:22:50.452270 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:50.451799 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tnr7k" podStartSLOduration=2.012260024 podStartE2EDuration="7.451777725s" totalImagesPullingTime="5.439517701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:50.450640584 +0000 UTC m=+8.705442708" watchObservedRunningTime="2026-11-01 22:22:50.451777725 +0000 UTC m=+8.706579854" Nov 01 22:22:51.289476 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.289336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:51.289652 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.289582 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:51.289782 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.289715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:51.289888 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.289835 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:51.289950 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.289912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:51.290071 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.290005 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:51.443326 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.443254 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" event={"ID":"0f3425f3-64ee-4134-b096-10e48c0c5faa","Type":"ContainerStarted","Data":"24667bdc476461608b5bc9c2bc4c6648b945e1aa5f0af75bc6b522c8de223ed7"} Nov 01 22:22:51.802683 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.802572 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:51.803483 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.803457 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:51.864355 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.864313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:51.864527 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.864371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:51.864573 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.864542 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:51.864623 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.864613 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs podName:47eb4322-3268-4e50-80e0-35f2dd802b0b nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.86459188 +0000 UTC m=+18.119393985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs") pod "network-metrics-daemon-x44vh" (UID: "47eb4322-3268-4e50-80e0-35f2dd802b0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:51.865536 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.865512 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:51.865682 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.865572 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret podName:7caa0db8-4f8f-4848-a2a6-845f20c9a888 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.865555305 +0000 UTC m=+18.120357411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret") pod "global-pull-secret-syncer-wdkm9" (UID: "7caa0db8-4f8f-4848-a2a6-845f20c9a888") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:51.965836 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:51.965801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:51.966022 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.965954 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:22:51.966022 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.965971 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:22:51.966022 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.965981 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwz2q for pod openshift-network-diagnostics/network-check-target-mpjqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:51.966189 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:51.966031 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q podName:3d4713d5-cc55-4d7e-86f8-f1e17f3b3858 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.966018217 +0000 UTC m=+18.220820318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwz2q" (UniqueName: "kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q") pod "network-check-target-mpjqm" (UID: "3d4713d5-cc55-4d7e-86f8-f1e17f3b3858") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:22:52.407697 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:52.407656 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:22:52.448763 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:52.448692 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" event={"ID":"0f3425f3-64ee-4134-b096-10e48c0c5faa","Type":"ContainerStarted","Data":"ac5aa93f41a26a6e353b1dd8e92baf8145daeb844495bce13e588adbfbb2108c"} Nov 01 22:22:53.288718 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.288513 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:53.288846 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.288577 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:53.288846 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:53.288833 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:53.288931 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.288598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:53.288931 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:53.288892 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:53.289023 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:53.288996 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:53.299950 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.299827 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:22:52.407677039Z","UUID":"3d64942b-c8f7-4d89-b5f9-206d05a2cd23","Handler":null,"Name":"","Endpoint":""} Nov 01 22:22:53.301676 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.301656 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:22:53.301788 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.301687 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:22:53.452493 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.452415 2578 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="7ad1d40b3e8b2dd6aa15cf6cbd13eab1a001acb962669d367c935649d3005186" exitCode=0 Nov 01 22:22:53.452887 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.452525 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerDied","Data":"7ad1d40b3e8b2dd6aa15cf6cbd13eab1a001acb962669d367c935649d3005186"} Nov 01 22:22:53.454506 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.454481 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n8wtc" event={"ID":"948069e6-a5b8-407d-a12b-198a76a2e783","Type":"ContainerStarted","Data":"2fdcdcb482e4d09b57ec7b8bcca78d0ae7067f24fb2eb5c7492bc0eb833f231b"} Nov 01 22:22:53.456410 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.456354 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gxf42" event={"ID":"aa89357c-1284-4f7a-b319-fcdb18edb560","Type":"ContainerStarted","Data":"9a8e2c31c43a15507bc9b42d0d399a98dae0df2d5053234057741c99334d3a14"} Nov 01 22:22:53.508621 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.508560 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-n8wtc" podStartSLOduration=2.286949956 podStartE2EDuration="10.50854261s" totalImagesPullingTime="8.221592654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:53.508170171 +0000 UTC m=+11.762972295" watchObservedRunningTime="2026-11-01 22:22:53.50854261 +0000 UTC m=+11.763344734" Nov 01 22:22:53.509121 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:53.509086 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-gxf42" podStartSLOduration=2.551974168 podStartE2EDuration="10.509071544s" totalImagesPullingTime="7.957097376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:53.493871675 +0000 UTC m=+11.748673800" watchObservedRunningTime="2026-11-01 22:22:53.509071544 +0000 UTC m=+11.763873671" Nov 01 22:22:54.460231 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:54.460126 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwft6" event={"ID":"7eeac783-1128-4402-8404-5962939c1b63","Type":"ContainerStarted","Data":"b02470a930f704b3181e842f977c4a811a45024b2ef14fe7dce7d97c01023eab"} Nov 01 22:22:54.462959 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:54.462927 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" event={"ID":"0f3425f3-64ee-4134-b096-10e48c0c5faa","Type":"ContainerStarted","Data":"a1d111a6fc0c78c5917af966a24f4133774cf36aa3974a34ce4b08ac2fc71901"} Nov 01 22:22:54.475199 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:54.475141 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bwft6" podStartSLOduration=2.955684792 podStartE2EDuration="11.475121615s" totalImagesPullingTime="8.519436823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:54.4747685 +0000 UTC m=+12.729570624" watchObservedRunningTime="2026-11-01 22:22:54.475121615 +0000 UTC m=+12.729923740" Nov 01 22:22:54.491870 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:54.491815 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" podStartSLOduration=2.272178452 podStartE2EDuration="11.491800964s" totalImagesPullingTime="9.219622512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:54.491097341 +0000 UTC m=+12.745899453" watchObservedRunningTime="2026-11-01 22:22:54.491800964 +0000 UTC m=+12.746603087" Nov 01 22:22:55.055484 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.055403 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:55.055681 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.055572 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:22:55.056332 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.056307 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:22:55.289128 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.289100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:55.289266 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.289153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:55.289266 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.289244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:55.289266 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:55.289245 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:55.289385 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:55.289309 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:55.289385 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:55.289371 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:55.466658 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.466561 2578 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="4ce0763e44b7c1c1cb19e8cb9c24e79b422df507c449274f6033c4f1ed3d51a7" exitCode=0 Nov 01 22:22:55.466658 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.466642 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerDied","Data":"4ce0763e44b7c1c1cb19e8cb9c24e79b422df507c449274f6033c4f1ed3d51a7"} Nov 01 22:22:55.468052 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.468030 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wrllx" event={"ID":"548c3a65-3217-4609-b266-b7eacca269a3","Type":"ContainerStarted","Data":"0b79eb63795eff62bf29949b3b660457f2818a511c27546bcc3f27e5cf5a8721"} Nov 01 22:22:55.502199 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:55.502156 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wrllx" podStartSLOduration=2.163755195 podStartE2EDuration="12.50214526s" totalImagesPullingTime="10.338390065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:22:55.501979284 +0000 UTC m=+13.756781404" watchObservedRunningTime="2026-11-01 22:22:55.50214526 +0000 UTC m=+13.756947425" Nov 01 22:22:57.288906 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:57.288731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:57.289357 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:57.288805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:57.289357 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:57.288986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:57.289357 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:57.289083 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:57.289357 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:57.288845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:57.289357 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:57.289163 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:57.473955 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:57.473925 2578 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="5d62e6b7ecb22c9e9a9a2a1154403d70ac1c5bb565361f026153a74ff072f4f2" exitCode=0 Nov 01 22:22:57.474099 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:57.473971 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerDied","Data":"5d62e6b7ecb22c9e9a9a2a1154403d70ac1c5bb565361f026153a74ff072f4f2"} Nov 01 22:22:59.289201 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:59.289168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:59.289644 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:59.289168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:22:59.289644 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:59.289295 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:22:59.289644 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:59.289168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:59.289644 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:59.289344 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:22:59.289644 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:59.289465 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:22:59.929171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:59.929149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:22:59.929277 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:22:59.929181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:22:59.929355 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:59.929334 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:59.929386 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:59.929355 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:22:59.929423 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:59.929408 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret podName:7caa0db8-4f8f-4848-a2a6-845f20c9a888 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:15.92938883 +0000 UTC m=+34.184190949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret") pod "global-pull-secret-syncer-wdkm9" (UID: "7caa0db8-4f8f-4848-a2a6-845f20c9a888") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:22:59.929460 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:22:59.929428 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs podName:47eb4322-3268-4e50-80e0-35f2dd802b0b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:15.929419001 +0000 UTC m=+34.184221116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs") pod "network-metrics-daemon-x44vh" (UID: "47eb4322-3268-4e50-80e0-35f2dd802b0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:23:00.029965 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:00.029938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:00.030107 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:00.030079 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:23:00.030107 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:00.030095 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:23:00.030107 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:00.030104 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwz2q for pod openshift-network-diagnostics/network-check-target-mpjqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:23:00.030210 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:00.030150 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q podName:3d4713d5-cc55-4d7e-86f8-f1e17f3b3858 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:16.030136484 +0000 UTC m=+34.284938588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwz2q" (UniqueName: "kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q") pod "network-check-target-mpjqm" (UID: "3d4713d5-cc55-4d7e-86f8-f1e17f3b3858") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:23:00.483265 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:00.483231 2578 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="bde6f6c58862ccddfa92735918e75429cf7f99905db3ceb09597442ffcc7c497" exitCode=0 Nov 01 22:23:00.483648 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:00.483273 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerDied","Data":"bde6f6c58862ccddfa92735918e75429cf7f99905db3ceb09597442ffcc7c497"} Nov 01 22:23:01.288767 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:01.288733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:01.288953 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:01.288808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:01.288953 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:01.288913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:01.288953 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:01.288932 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:01.289097 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:01.288975 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:01.289097 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:01.289036 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:01.488366 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:01.488333 2578 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="bee4432288650de90c638be26dcc614d8d0d8dbbce6c9736319cb9a9f50867eb" exitCode=0 Nov 01 22:23:01.488767 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:01.488383 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerDied","Data":"bee4432288650de90c638be26dcc614d8d0d8dbbce6c9736319cb9a9f50867eb"} Nov 01 22:23:02.494946 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:02.494913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" event={"ID":"a4ce9767-19f2-42a8-afe1-ad361726b8b9","Type":"ContainerStarted","Data":"cf21d5ec206760b0a0e544bee21761d76fb724e8993de510eff3821e5d066de2"} Nov 01 22:23:02.518119 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:02.518070 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bbtw6" podStartSLOduration=7.066643224 podStartE2EDuration="19.518056432s" totalImagesPullingTime="12.451413208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:02.516311768 +0000 UTC m=+20.771113885" watchObservedRunningTime="2026-11-01 22:23:02.518056432 +0000 UTC m=+20.772858551" Nov 01 22:23:03.288332 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:03.288298 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:03.288507 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:03.288353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:03.288507 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:03.288465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:03.288507 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:03.288470 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:03.288663 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:03.288604 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:03.288712 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:03.288668 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:05.288826 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:05.288783 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:05.288826 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:05.288809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:05.288826 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:05.288829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:05.289380 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:05.288940 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:05.289380 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:05.289036 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:05.289380 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:05.289131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:07.288902 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.288676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:07.289439 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.288703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:07.289439 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:07.288931 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:07.289439 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.288711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:07.289439 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:07.288983 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:07.289439 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:07.289072 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:07.508349 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.508320 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"8dfe59a014f54753447fceb08be0fdbd8b233ba270cce7374d9406453abfea16"} Nov 01 22:23:07.508498 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.508382 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"de4f534d73034166b367efd9785b5bf7e5f047bd7c8ebb4f1b357d41a5b3d9bf"} Nov 01 22:23:07.508498 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.508414 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"3f3a4829bdef63bfef08def07d3805f4e5a73adbd146cfa1c766014f9282c0ee"} Nov 01 22:23:07.508498 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.508446 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"eeb392a57c50542962bc4293af423a6a7d8bb6b5dea7917a112cb84fe003e1d6"} Nov 01 22:23:07.508498 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.508472 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"bef9d92c1c0136044cdfc5acc9cda9c1e386995b3dc7cf2bba6ee4a2d7a64578"} Nov 01 22:23:07.508630 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:07.508503 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"6f81fa6b2029fe9e63b9952c5bb60a6eb587d04e2d84b3666f92be77f0712f2b"} Nov 01 22:23:09.288896 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:09.288869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:09.289434 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:09.288907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:09.289434 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:09.288911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:09.289434 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:09.288990 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:09.289434 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:09.289050 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:09.289434 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:09.289125 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:09.514554 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:09.514518 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"4244179556af18f96ad15e19af831bc6a6073453737939953cc674a38c320515"} Nov 01 22:23:11.288544 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.288300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:11.289196 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.288337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:11.289196 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:11.288643 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:11.289196 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.288378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:11.289196 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:11.288694 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:11.289196 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:11.288750 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:11.522742 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.522616 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" event={"ID":"e91e58ba-dbc6-49da-b601-22edc7b6525e","Type":"ContainerStarted","Data":"03af7dbb6c79cad14687566e0c80fdf106d4ece43174a23fa8407fa2678cea6a"} Nov 01 22:23:11.522962 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.522933 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:11.523098 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.522996 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:11.537951 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.537927 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:11.538115 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.538048 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:11.551390 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:11.551355 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" podStartSLOduration=6.255881204 podStartE2EDuration="28.551344619s" totalImagesPullingTime="22.295463415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:11.550978218 +0000 UTC m=+29.805780337" watchObservedRunningTime="2026-11-01 22:23:11.551344619 +0000 UTC m=+29.806146738" Nov 01 22:23:12.524958 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:12.524931 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:23:12.901759 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:12.901675 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdkm9"] Nov 01 22:23:12.901934 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:12.901848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:12.902006 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:12.901958 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:12.904739 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:12.904711 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mpjqm"] Nov 01 22:23:12.904874 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:12.904821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:12.904933 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:12.904915 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:12.905336 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:12.905315 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x44vh"] Nov 01 22:23:12.905438 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:12.905413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:12.905536 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:12.905514 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:13.526394 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:13.526367 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:23:14.157161 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:14.157132 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:14.277234 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:14.277202 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:14.289172 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:14.289137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:14.289276 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:14.289147 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:14.289276 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:14.289255 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:14.289356 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:14.289303 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:14.289356 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:14.289157 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:14.289414 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:14.289388 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:14.370091 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:14.370064 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:15.946732 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:15.946697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:15.946732 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:15.946738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:15.947165 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:15.946839 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 22:23:15.947165 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:15.946851 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:23:15.947165 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:15.946906 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret podName:7caa0db8-4f8f-4848-a2a6-845f20c9a888 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.946891819 +0000 UTC m=+66.201693917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret") pod "global-pull-secret-syncer-wdkm9" (UID: "7caa0db8-4f8f-4848-a2a6-845f20c9a888") : object "kube-system"/"original-pull-secret" not registered Nov 01 22:23:15.947165 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:15.946920 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs podName:47eb4322-3268-4e50-80e0-35f2dd802b0b nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.946913784 +0000 UTC m=+66.201715882 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs") pod "network-metrics-daemon-x44vh" (UID: "47eb4322-3268-4e50-80e0-35f2dd802b0b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 22:23:16.047967 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:16.047930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:16.048130 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:16.048066 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 22:23:16.048130 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:16.048080 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 22:23:16.048130 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:16.048088 2578 projected.go:196] Error preparing data for projected volume kube-api-access-wwz2q for pod openshift-network-diagnostics/network-check-target-mpjqm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:23:16.048225 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:16.048134 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q podName:3d4713d5-cc55-4d7e-86f8-f1e17f3b3858 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:48.04812179 +0000 UTC m=+66.302923901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wwz2q" (UniqueName: "kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q") pod "network-check-target-mpjqm" (UID: "3d4713d5-cc55-4d7e-86f8-f1e17f3b3858") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 22:23:16.288752 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:16.288727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:16.288752 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:16.288741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:16.288987 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:16.288727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:16.288987 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:16.288819 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:16.288987 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:16.288893 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:16.288987 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:16.288959 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:16.681251 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:16.681220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:18.289247 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:18.289209 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:18.289247 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:18.289235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:18.289688 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:18.289315 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:18.289688 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:18.289330 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wdkm9" podUID="7caa0db8-4f8f-4848-a2a6-845f20c9a888" Nov 01 22:23:18.289688 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:18.289449 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x44vh" podUID="47eb4322-3268-4e50-80e0-35f2dd802b0b" Nov 01 22:23:18.289688 ip-10-0-138-87 kubenswrapper[2578]: E1101 22:23:18.289505 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mpjqm" podUID="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" Nov 01 22:23:18.490190 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:18.490156 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:18.601640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:18.601552 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:18.690310 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:18.690280 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:18.905383 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:18.905313 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:19.461990 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:19.461946 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:19.557397 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:19.557327 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:19.657715 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:19.657688 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:20.016606 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.016417 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeReady" Nov 01 22:23:20.016763 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.016721 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 22:23:20.288922 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.288884 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:20.288922 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.288909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:20.289122 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.289019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:20.293881 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.293859 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:20.294004 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.293899 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:20.294140 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.294125 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:20.295486 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.295465 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:20.295593 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.295505 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:20.295593 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.295522 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:20.295668 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.295569 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:20.598502 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.598422 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:20.679327 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:20.679293 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:22.224216 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:22.224173 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:22.622214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:22.622188 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:22.671587 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:22.671556 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n8wtc_948069e6-a5b8-407d-a12b-198a76a2e783/dns-node-resolver/0.log" Nov 01 22:23:22.720815 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:22.720791 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:22.821031 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:22.821002 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:22.921123 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:22.921057 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:23.871420 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:23.871392 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnr7k_f130f8c5-0161-4ef2-8634-cc150d09ddf0/node-ca/0.log" Nov 01 22:23:25.537847 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:25.537816 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:25.644035 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:25.643999 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:25.748170 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:25.748128 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:25.824214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:25.824154 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:25.982601 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:25.982561 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.098169 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:26.098101 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.198919 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:26.198886 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.447157 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:26.447115 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:26.653735 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:26.653697 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.185094 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:27.185064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.266374 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:27.266342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.373926 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:27.373891 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.572981 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:27.572900 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.684385 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:27.684355 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.784236 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:27.784208 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:27.981159 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:27.981129 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:28.217323 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:28.217296 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:28.319783 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:28.319712 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:28.418357 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:28.418328 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:28.520245 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:28.520213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:28.640664 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:28.640564 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:29.149907 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:29.149876 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:29.250246 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:29.250211 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:29.381353 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:29.381290 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:29.539065 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:29.539035 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:29.639898 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:29.639862 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:29.784382 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:29.784354 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:29.971404 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:29.971374 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:30.911659 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:30.911628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:31.172715 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:31.172684 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:31.386844 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:31.386814 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:31.572090 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:31.572010 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:31.775919 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:31.775873 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:31.978427 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:31.978395 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:32.145101 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:32.145072 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.172194 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:33.172159 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.370438 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:33.370413 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.574665 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:33.574626 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.782100 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:33.782067 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:33.984090 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:33.984064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:35.183171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:35.183139 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:35.377979 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:35.377948 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:35.572592 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:35.572459 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:35.673079 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:35.673049 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:35.793886 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:35.793853 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:35.985246 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:35.985222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:36.397008 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:36.396916 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:40.196987 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.196954 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:40.282425 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.282395 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-82z6g"] Nov 01 22:23:40.289637 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.289614 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.292652 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.292630 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-n87cr"] Nov 01 22:23:40.295768 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.295715 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5v8l4"] Nov 01 22:23:40.296066 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.296044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.297199 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.297166 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:40.298871 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.298848 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-82z6g"] Nov 01 22:23:40.298963 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.298938 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.301015 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.301000 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:40.308623 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.308603 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:40.308623 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.308620 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhd78\"" Nov 01 22:23:40.311773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.310614 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-n87cr"] Nov 01 22:23:40.311858 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.311775 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5v8l4"] Nov 01 22:23:40.324888 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.324872 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m9kr7"] Nov 01 22:23:40.327806 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.327792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.334164 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.334132 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 22:23:40.334415 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.334401 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:40.334460 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.334407 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 22:23:40.341170 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.341154 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m9kr7"] Nov 01 22:23:40.343797 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.343779 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:40.343897 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.343848 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:40.343897 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.343863 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:40.374441 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.374415 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d7bc7f98b-p87lt"] Nov 01 22:23:40.377281 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.377268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.391092 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.391074 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 22:23:40.391843 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.391823 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d7bc7f98b-p87lt"] Nov 01 22:23:40.391970 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.391946 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:40.397813 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.397796 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:40.397813 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.397803 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 22:23:40.398778 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.398762 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 22:23:40.398833 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.398780 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6kgdt\"" Nov 01 22:23:40.398833 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.398762 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:40.399015 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.399001 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:40.399015 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.399010 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bpr7j\"" Nov 01 22:23:40.399080 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.399008 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 22:23:40.399080 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.399021 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:40.410396 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2w7\" (UniqueName: \"kubernetes.io/projected/874207d9-9be6-4544-bc12-2bf935d7e477-kube-api-access-2w2w7\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.410472 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e15bc3a-dc26-4b68-bba9-b670d741c952-cert\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.410472 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/874207d9-9be6-4544-bc12-2bf935d7e477-config-volume\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.410533 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s28ks\" (UniqueName: \"kubernetes.io/projected/8deca68d-1a0b-4841-9c91-0682262d59c7-kube-api-access-s28ks\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.410570 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46gn\" (UniqueName: \"kubernetes.io/projected/4e15bc3a-dc26-4b68-bba9-b670d741c952-kube-api-access-z46gn\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.410668 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/874207d9-9be6-4544-bc12-2bf935d7e477-metrics-tls\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.410668 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410606 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deca68d-1a0b-4841-9c91-0682262d59c7-tmp\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.410668 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.410623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/874207d9-9be6-4544-bc12-2bf935d7e477-tmp-dir\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.490020 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.489956 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:40.511895 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.511873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/874207d9-9be6-4544-bc12-2bf935d7e477-tmp-dir\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.511972 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.511908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2w7\" (UniqueName: \"kubernetes.io/projected/874207d9-9be6-4544-bc12-2bf935d7e477-kube-api-access-2w2w7\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.511972 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.511928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-trusted-ca\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512075 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512057 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.512113 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.512217 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e15bc3a-dc26-4b68-bba9-b670d741c952-cert\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.512254 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-image-registry-private-configuration\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512254 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512237 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/874207d9-9be6-4544-bc12-2bf935d7e477-tmp-dir\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.512254 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/874207d9-9be6-4544-bc12-2bf935d7e477-config-volume\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.512344 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28ks\" (UniqueName: \"kubernetes.io/projected/8deca68d-1a0b-4841-9c91-0682262d59c7-kube-api-access-s28ks\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.512397 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-installation-pull-secrets\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512430 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46gn\" (UniqueName: \"kubernetes.io/projected/4e15bc3a-dc26-4b68-bba9-b670d741c952-kube-api-access-z46gn\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.512465 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/874207d9-9be6-4544-bc12-2bf935d7e477-metrics-tls\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.512465 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-crio-socket\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.512570 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-data-volume\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.512650 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8f9\" (UniqueName: \"kubernetes.io/projected/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-api-access-dm8f9\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.512650 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deca68d-1a0b-4841-9c91-0682262d59c7-tmp\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.512738 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83ec6a1b-db92-4103-adec-8539c8bb86c2-ca-trust-extracted\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512738 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-certificates\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512836 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-tls\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512836 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-bound-sa-token\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512836 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km45f\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-kube-api-access-km45f\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.512836 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/874207d9-9be6-4544-bc12-2bf935d7e477-config-volume\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.512979 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.512945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deca68d-1a0b-4841-9c91-0682262d59c7-tmp\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.515894 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.515875 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/874207d9-9be6-4544-bc12-2bf935d7e477-metrics-tls\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.515894 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.515892 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e15bc3a-dc26-4b68-bba9-b670d741c952-cert\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.520907 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.520888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46gn\" (UniqueName: \"kubernetes.io/projected/4e15bc3a-dc26-4b68-bba9-b670d741c952-kube-api-access-z46gn\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.525783 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.525756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2w7\" (UniqueName: \"kubernetes.io/projected/874207d9-9be6-4544-bc12-2bf935d7e477-kube-api-access-2w2w7\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.525853 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.525804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28ks\" (UniqueName: \"kubernetes.io/projected/8deca68d-1a0b-4841-9c91-0682262d59c7-kube-api-access-s28ks\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.600882 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.600860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:40.609985 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.609961 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:40.613879 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.613857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-trusted-ca\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.613952 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.613894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.613952 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.613920 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.614073 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.613994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-image-registry-private-configuration\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614073 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-installation-pull-secrets\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614073 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-crio-socket\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.614214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-data-volume\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.614214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8f9\" (UniqueName: \"kubernetes.io/projected/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-api-access-dm8f9\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.614214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614129 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83ec6a1b-db92-4103-adec-8539c8bb86c2-ca-trust-extracted\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614153 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-certificates\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-tls\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614214 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-bound-sa-token\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614464 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km45f\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-kube-api-access-km45f\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614557 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614535 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.614817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-data-volume\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.614817 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-trusted-ca\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.614951 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.614877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83ec6a1b-db92-4103-adec-8539c8bb86c2-ca-trust-extracted\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.615270 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.615187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-certificates\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.615532 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.615503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-crio-socket\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.617165 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.617134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-installation-pull-secrets\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.617165 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.617153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.617310 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.617143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-image-registry-private-configuration\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.617796 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.617778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-tls\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.619226 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.619202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:40.623702 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.623653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-bound-sa-token\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.624091 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.624073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km45f\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-kube-api-access-km45f\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.624704 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.624679 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8f9\" (UniqueName: \"kubernetes.io/projected/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-api-access-dm8f9\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.642314 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.642282 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:40.687461 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.687168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:40.888317 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.888167 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-82z6g"] Nov 01 22:23:40.892563 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.892536 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-n87cr"] Nov 01 22:23:40.895821 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.895804 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5v8l4"] Nov 01 22:23:40.896026 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:40.896002 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deca68d_1a0b_4841_9c91_0682262d59c7.slice/crio-0a6fcdb1ebaaaed5ab5e6f1063eff317ff584f1d208faaffea90a551194c85e1 WatchSource:0}: Error finding container 0a6fcdb1ebaaaed5ab5e6f1063eff317ff584f1d208faaffea90a551194c85e1: Status 404 returned error can't find the container with id 0a6fcdb1ebaaaed5ab5e6f1063eff317ff584f1d208faaffea90a551194c85e1 Nov 01 22:23:40.951094 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.951063 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m9kr7"] Nov 01 22:23:40.953011 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:40.952991 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cf0fb4_ef5d_4921_9c01_8830c386dce8.slice/crio-bd68110155231b38a3b26e68d7fdaa9ae06512c587ba785601601c3d9dfc6d0a WatchSource:0}: Error finding container bd68110155231b38a3b26e68d7fdaa9ae06512c587ba785601601c3d9dfc6d0a: Status 404 returned error can't find the container with id bd68110155231b38a3b26e68d7fdaa9ae06512c587ba785601601c3d9dfc6d0a Nov 01 22:23:40.966124 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:40.966103 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d7bc7f98b-p87lt"] Nov 01 22:23:40.979411 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:40.979391 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ec6a1b_db92_4103_adec_8539c8bb86c2.slice/crio-2a0364f0eaa5303dc1e100cd28a83623349d7f83c9666c8b73693600baeafd79 WatchSource:0}: Error finding container 2a0364f0eaa5303dc1e100cd28a83623349d7f83c9666c8b73693600baeafd79: Status 404 returned error can't find the container with id 2a0364f0eaa5303dc1e100cd28a83623349d7f83c9666c8b73693600baeafd79 Nov 01 22:23:41.592794 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.592727 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5v8l4" event={"ID":"874207d9-9be6-4544-bc12-2bf935d7e477","Type":"ContainerStarted","Data":"8022477da90b6752f2e4d3d8c7c045597dcf42c2cd6a37833f85355db84dc869"} Nov 01 22:23:41.594939 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.594883 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-82z6g" event={"ID":"4e15bc3a-dc26-4b68-bba9-b670d741c952","Type":"ContainerStarted","Data":"abefdd856761d8cb798c3204eb491c6d0f3e4a47d038e9823b916b0772ac638e"} Nov 01 22:23:41.596966 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.596890 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9kr7" event={"ID":"d9cf0fb4-ef5d-4921-9c01-8830c386dce8","Type":"ContainerStarted","Data":"e8e12bdcfe5f063f3ccb3a970df8da5b9bf660cc4810bcec8dff76737138074f"} Nov 01 22:23:41.596966 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.596945 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9kr7" event={"ID":"d9cf0fb4-ef5d-4921-9c01-8830c386dce8","Type":"ContainerStarted","Data":"bd68110155231b38a3b26e68d7fdaa9ae06512c587ba785601601c3d9dfc6d0a"} Nov 01 22:23:41.598639 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.598615 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-n87cr" event={"ID":"8deca68d-1a0b-4841-9c91-0682262d59c7","Type":"ContainerStarted","Data":"0a6fcdb1ebaaaed5ab5e6f1063eff317ff584f1d208faaffea90a551194c85e1"} Nov 01 22:23:41.600631 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.600606 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" event={"ID":"83ec6a1b-db92-4103-adec-8539c8bb86c2","Type":"ContainerStarted","Data":"2991c0d35c089feee909ef93bd69fbd43a3b4bedeb5b6dd5d81a91d2113de3a3"} Nov 01 22:23:41.600746 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.600649 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" event={"ID":"83ec6a1b-db92-4103-adec-8539c8bb86c2","Type":"ContainerStarted","Data":"2a0364f0eaa5303dc1e100cd28a83623349d7f83c9666c8b73693600baeafd79"} Nov 01 22:23:41.600802 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.600761 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:41.622423 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:41.622371 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" podStartSLOduration=1.622354331 podStartE2EDuration="1.622354331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:41.620731536 +0000 UTC m=+59.875533658" watchObservedRunningTime="2026-11-01 22:23:41.622354331 +0000 UTC m=+59.877156452" Nov 01 22:23:42.606104 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:42.606060 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9kr7" event={"ID":"d9cf0fb4-ef5d-4921-9c01-8830c386dce8","Type":"ContainerStarted","Data":"4d8e0e96474fba65b3da8edce3b4548557ffd0dbbee141d47a050be4c4e51da2"} Nov 01 22:23:42.741328 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:42.741288 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:42.820031 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:42.819998 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:44.539680 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:44.539618 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:44.612299 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:44.612273 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-82z6g" event={"ID":"4e15bc3a-dc26-4b68-bba9-b670d741c952","Type":"ContainerStarted","Data":"d08c2ec9cdf5f6633ee61d4b752d914d416ad99756454c8c90303f60829a1dc9"} Nov 01 22:23:44.628007 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:44.627952 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-82z6g" podStartSLOduration=1.26296729 podStartE2EDuration="4.627935321s" totalImagesPullingTime="3.364968031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:44.627493349 +0000 UTC m=+62.882295469" watchObservedRunningTime="2026-11-01 22:23:44.627935321 +0000 UTC m=+62.882737441" Nov 01 22:23:45.617839 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:45.617799 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5v8l4" event={"ID":"874207d9-9be6-4544-bc12-2bf935d7e477","Type":"ContainerStarted","Data":"9f1dd4eba3f524ae3d45ce4cc0727b2cb1ebe5e0b13396d9ae08c53a7d6bdbb7"} Nov 01 22:23:45.618276 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:45.617895 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:45.618276 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:45.617914 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5v8l4" event={"ID":"874207d9-9be6-4544-bc12-2bf935d7e477","Type":"ContainerStarted","Data":"40258df58c4fe9abc4b4810468bdb41e6ea437a2b824c9580f49cf8450a96197"} Nov 01 22:23:45.620321 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:45.620289 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9kr7" event={"ID":"d9cf0fb4-ef5d-4921-9c01-8830c386dce8","Type":"ContainerStarted","Data":"7c622a10a46fc48d6ee76aa5a05fe19b146bc5d89706c8b26056c1fe19a1854d"} Nov 01 22:23:45.638289 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:45.638241 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5v8l4" podStartSLOduration=1.942388064 podStartE2EDuration="5.638222493s" totalImagesPullingTime="3.695834429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:45.636271283 +0000 UTC m=+63.891073414" watchObservedRunningTime="2026-11-01 22:23:45.638222493 +0000 UTC m=+63.893024614" Nov 01 22:23:45.656052 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:45.656016 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m9kr7" podStartSLOduration=2.122992504 podStartE2EDuration="5.656004651s" totalImagesPullingTime="3.533012147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:23:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:45.654976982 +0000 UTC m=+63.909779104" watchObservedRunningTime="2026-11-01 22:23:45.656004651 +0000 UTC m=+63.910806771" Nov 01 22:23:45.977455 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:45.977415 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:46.081008 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.080972 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:46.188421 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.188391 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp"] Nov 01 22:23:46.191522 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.191501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:46.194889 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.194867 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 22:23:46.195085 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.195070 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vfgt9\"" Nov 01 22:23:46.199202 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.199178 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp"] Nov 01 22:23:46.206089 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.206065 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:46.258210 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.258144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c512618-f7bd-4210-90f9-e525e4a8a1b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2k4xp\" (UID: \"5c512618-f7bd-4210-90f9-e525e4a8a1b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:46.359521 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.359481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c512618-f7bd-4210-90f9-e525e4a8a1b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2k4xp\" (UID: \"5c512618-f7bd-4210-90f9-e525e4a8a1b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:46.362296 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.362245 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c512618-f7bd-4210-90f9-e525e4a8a1b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2k4xp\" (UID: \"5c512618-f7bd-4210-90f9-e525e4a8a1b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:46.383126 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.383096 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:46.505219 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.505186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:46.772475 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:46.772335 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp"] Nov 01 22:23:46.774738 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:46.774712 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c512618_f7bd_4210_90f9_e525e4a8a1b7.slice/crio-9629a8c8bb23451632cf19c51d7d9fe02cd060bbd708d7918faee8f7659a7222 WatchSource:0}: Error finding container 9629a8c8bb23451632cf19c51d7d9fe02cd060bbd708d7918faee8f7659a7222: Status 404 returned error can't find the container with id 9629a8c8bb23451632cf19c51d7d9fe02cd060bbd708d7918faee8f7659a7222 Nov 01 22:23:47.628611 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:47.628558 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" event={"ID":"5c512618-f7bd-4210-90f9-e525e4a8a1b7","Type":"ContainerStarted","Data":"9629a8c8bb23451632cf19c51d7d9fe02cd060bbd708d7918faee8f7659a7222"} Nov 01 22:23:47.973982 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:47.973945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:47.974341 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:47.973996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:47.978298 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:47.978279 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:47.978399 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:47.978302 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:47.987165 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:47.987096 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:47.987165 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:47.987104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:48.074767 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.074730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:48.077558 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.077530 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:48.088061 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.088037 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:48.098846 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.098820 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:48.218761 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.218732 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:48.226822 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.226802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:48.244586 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.244554 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:48.252645 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.252625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:48.258270 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.258251 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:48.266357 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.266336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:48.502450 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.502379 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wdkm9"] Nov 01 22:23:48.505040 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:48.505014 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caa0db8_4f8f_4848_a2a6_845f20c9a888.slice/crio-e4c9d31ef980c288441b066a4dc7a8a92a6e682b6c8ee49bc8ae4a05e329ff89 WatchSource:0}: Error finding container e4c9d31ef980c288441b066a4dc7a8a92a6e682b6c8ee49bc8ae4a05e329ff89: Status 404 returned error can't find the container with id e4c9d31ef980c288441b066a4dc7a8a92a6e682b6c8ee49bc8ae4a05e329ff89 Nov 01 22:23:48.541828 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.541793 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x44vh"] Nov 01 22:23:48.544553 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.544531 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mpjqm"] Nov 01 22:23:48.545267 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:48.545239 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47eb4322_3268_4e50_80e0_35f2dd802b0b.slice/crio-ec988848c8755e82f264fb0d5d373d798325e9d63842cc35b8054655a4664ca7 WatchSource:0}: Error finding container ec988848c8755e82f264fb0d5d373d798325e9d63842cc35b8054655a4664ca7: Status 404 returned error can't find the container with id ec988848c8755e82f264fb0d5d373d798325e9d63842cc35b8054655a4664ca7 Nov 01 22:23:48.548288 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:48.548262 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4713d5_cc55_4d7e_86f8_f1e17f3b3858.slice/crio-f09a1075b225f6706e04a61bb03794e5a73bd0a3ce2490386d2d4d11e660539f WatchSource:0}: Error finding container f09a1075b225f6706e04a61bb03794e5a73bd0a3ce2490386d2d4d11e660539f: Status 404 returned error can't find the container with id f09a1075b225f6706e04a61bb03794e5a73bd0a3ce2490386d2d4d11e660539f Nov 01 22:23:48.633557 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.633510 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mpjqm" event={"ID":"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858","Type":"ContainerStarted","Data":"f09a1075b225f6706e04a61bb03794e5a73bd0a3ce2490386d2d4d11e660539f"} Nov 01 22:23:48.634773 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.634739 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wdkm9" event={"ID":"7caa0db8-4f8f-4848-a2a6-845f20c9a888","Type":"ContainerStarted","Data":"e4c9d31ef980c288441b066a4dc7a8a92a6e682b6c8ee49bc8ae4a05e329ff89"} Nov 01 22:23:48.635925 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.635898 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x44vh" event={"ID":"47eb4322-3268-4e50-80e0-35f2dd802b0b","Type":"ContainerStarted","Data":"ec988848c8755e82f264fb0d5d373d798325e9d63842cc35b8054655a4664ca7"} Nov 01 22:23:48.637355 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.637334 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" event={"ID":"5c512618-f7bd-4210-90f9-e525e4a8a1b7","Type":"ContainerStarted","Data":"7b80ed457af07e206fa655dd57ebcc97cb08219be54e6f12d2a5a1d5c1c90e19"} Nov 01 22:23:48.637617 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.637561 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:48.642943 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.642919 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:48.657065 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:48.657022 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" podStartSLOduration=1.585904293 podStartE2EDuration="2.657007477s" totalImagesPullingTime="1.071103184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:48.656956295 +0000 UTC m=+66.911758418" watchObservedRunningTime="2026-11-01 22:23:48.657007477 +0000 UTC m=+66.911809595" Nov 01 22:23:49.227285 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:49.227248 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:49.324977 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:49.324946 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:49.412807 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:49.412771 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:50.347775 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.347737 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85d9849f88-4ntzt"] Nov 01 22:23:50.352270 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.352246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.362608 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.362115 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 22:23:50.362608 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.362263 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 22:23:50.362608 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.362315 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 22:23:50.362608 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.362365 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j4zfh\"" Nov 01 22:23:50.362608 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.362404 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 22:23:50.362608 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.362432 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 22:23:50.364500 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.363916 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d9849f88-4ntzt"] Nov 01 22:23:50.393171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.392820 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.393171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.392859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.393171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.392890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.393171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.392935 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.393171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.392958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6gnk\" (UniqueName: \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.393171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.392982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.393171 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.393009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.493915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.493963 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gnk\" (UniqueName: \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.493993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.494022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.494062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.494092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.494124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.494898 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.495640 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.495471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.496262 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.496227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.496527 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.496508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.498273 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.498248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.498935 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.498913 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.504048 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.504025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gnk\" (UniqueName: \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.646767 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.646665 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x44vh" event={"ID":"47eb4322-3268-4e50-80e0-35f2dd802b0b","Type":"ContainerStarted","Data":"54e31df12bd1e3cc6207951adb8a94bddaa640bced4850efee30ecdf9ea43877"} Nov 01 22:23:50.646767 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.646728 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x44vh" event={"ID":"47eb4322-3268-4e50-80e0-35f2dd802b0b","Type":"ContainerStarted","Data":"5e1fd98fcdf63b2c80c9564d339864b3ee4eb5e8319018e39bf15abee6709569"} Nov 01 22:23:50.664376 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.664321 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x44vh" podStartSLOduration=66.012644743 podStartE2EDuration="1m7.664303707s" totalImagesPullingTime="1.651658964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:50.663251078 +0000 UTC m=+68.918053200" watchObservedRunningTime="2026-11-01 22:23:50.664303707 +0000 UTC m=+68.919105827" Nov 01 22:23:50.667878 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.667854 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:50.949268 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:50.949221 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d9849f88-4ntzt"] Nov 01 22:23:50.953942 ip-10-0-138-87 kubenswrapper[2578]: W1101 22:23:50.953906 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe7ab41_3bca_400b_9a55_ed06fb29c216.slice/crio-48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454 WatchSource:0}: Error finding container 48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454: Status 404 returned error can't find the container with id 48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454 Nov 01 22:23:51.653936 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:51.653867 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d9849f88-4ntzt" event={"ID":"2fe7ab41-3bca-400b-9a55-ed06fb29c216","Type":"ContainerStarted","Data":"48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454"} Nov 01 22:23:53.720443 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:53.720407 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:54.484926 ip-10-0-138-87 kubenswrapper[2578]: I1101 22:23:54.484874 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:54.484992 ip-10-0-138-87 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 22:23:54.505724 ip-10-0-138-87 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 22:23:54.505958 ip-10-0-138-87 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 22:23:54.506190 ip-10-0-138-87 systemd[1]: kubelet.service: Consumed 3.273s CPU time. Nov 01 22:23:54.514195 ip-10-0-138-87 systemd[1]: Starting Kubernetes Kubelet... Nov 01 22:23:54.619437 ip-10-0-138-87 kubenswrapper[4981]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:54.619437 ip-10-0-138-87 kubenswrapper[4981]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 22:23:54.619437 ip-10-0-138-87 kubenswrapper[4981]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:54.619437 ip-10-0-138-87 kubenswrapper[4981]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 22:23:54.622311 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622208 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.622311 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622311 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622318 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622322 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622326 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622332 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622337 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622341 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622346 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622352 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622356 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622360 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622364 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622377 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622382 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622386 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622391 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622395 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622400 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622405 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622410 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.622474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622414 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622418 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622423 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622427 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622431 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622435 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622439 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622443 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622448 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622454 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622458 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622463 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622468 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622473 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622477 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622482 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622486 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622490 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622494 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622500 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.623378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622504 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622509 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622513 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622517 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622522 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622529 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622533 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622538 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622542 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622546 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622549 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622553 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622560 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622567 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622586 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622593 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622599 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622605 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.624094 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622611 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622616 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622620 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622625 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622631 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622636 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622641 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622646 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622651 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622657 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622662 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622667 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622671 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622675 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622679 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622683 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622686 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622690 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622694 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.624750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622698 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622703 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622708 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622712 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622717 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622721 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622724 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622729 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622734 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622739 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622744 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622749 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622753 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622757 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622761 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622766 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622770 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622775 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622779 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622783 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.625384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622787 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622792 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622797 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622801 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622805 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622809 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622814 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.622818 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623899 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623910 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623915 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623920 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623924 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623928 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623935 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623940 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623945 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623950 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623954 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.626089 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623958 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623962 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623966 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623970 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623975 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623979 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623984 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623988 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623992 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.623996 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624000 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624005 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624008 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624013 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624016 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624020 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624024 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624029 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624036 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.626815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624040 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624045 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624049 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624053 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624058 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624062 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624067 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624070 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624075 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624079 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624083 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624087 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624092 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624096 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624101 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624105 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624111 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624117 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624121 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.627390 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624125 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624130 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624134 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624138 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624142 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624147 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624150 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624154 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624158 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624161 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624166 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624171 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624181 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624185 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624188 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624190 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624196 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624203 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624207 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624212 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.628172 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624216 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624220 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624225 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624230 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624234 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624239 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624243 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624247 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624251 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624255 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624259 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624264 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624269 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624273 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624278 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624282 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624286 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624290 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624294 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624299 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.629017 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624303 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624307 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624312 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624317 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624322 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624326 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624330 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624335 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624339 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624343 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624347 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624352 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624357 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624362 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624366 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624376 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.624381 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625119 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625130 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625136 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.629857 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625141 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625147 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625152 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625156 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625160 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625165 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625169 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625172 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625174 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625177 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625180 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625182 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625185 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625188 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625191 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625193 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625196 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625199 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625201 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.630538 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625204 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625206 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625209 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625211 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625214 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625217 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625219 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625222 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625227 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625231 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625234 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625237 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625239 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625242 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625244 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625247 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625249 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625252 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625254 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.631075 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625257 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625259 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625262 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625265 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625267 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625270 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625273 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625275 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625278 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625281 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625283 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625286 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625288 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625291 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625293 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625296 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625298 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625301 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625303 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625306 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.631568 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625309 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625312 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625315 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625319 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625321 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625324 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625326 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625329 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625331 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625334 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625336 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625340 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625343 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625345 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625348 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625350 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625353 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625355 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625358 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625362 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.632170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625366 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625369 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625374 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625378 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625383 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625388 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625392 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625397 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625400 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625403 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625406 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625408 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625411 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625415 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625417 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625420 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625423 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625427 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625429 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625432 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.632993 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625434 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625438 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625441 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625445 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.625447 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625633 4981 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625648 4981 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625657 4981 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625664 4981 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625671 4981 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625677 4981 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625683 4981 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625692 4981 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625697 4981 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625701 4981 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625707 4981 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625711 4981 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625717 4981 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625721 4981 flags.go:64] FLAG: --cgroup-root="" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625726 4981 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625732 4981 flags.go:64] FLAG: --client-ca-file="" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625736 4981 flags.go:64] FLAG: --cloud-provider="external" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625742 4981 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 22:23:54.633813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625749 4981 flags.go:64] FLAG: --cluster-domain="" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625753 4981 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625760 4981 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625764 4981 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625769 4981 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625775 4981 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625787 4981 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625792 4981 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625796 4981 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625800 4981 flags.go:64] FLAG: --contention-profiling="false" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625805 4981 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625810 4981 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625815 4981 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625820 4981 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625828 4981 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625831 4981 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625834 4981 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625837 4981 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625840 4981 flags.go:64] FLAG: --enable-server="true" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625843 4981 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625847 4981 flags.go:64] FLAG: --event-burst="100" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625851 4981 flags.go:64] FLAG: --event-qps="50" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625854 4981 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625857 4981 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 22:23:54.634793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625860 4981 flags.go:64] FLAG: --eviction-hard="" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625866 4981 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625869 4981 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625872 4981 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625875 4981 flags.go:64] FLAG: --eviction-soft="" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625878 4981 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625881 4981 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625884 4981 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625887 4981 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625889 4981 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625893 4981 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625896 4981 flags.go:64] FLAG: --feature-gates="" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625900 4981 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625904 4981 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625909 4981 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625914 4981 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625920 4981 flags.go:64] FLAG: --healthz-port="10248" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625925 4981 flags.go:64] FLAG: --help="false" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625930 4981 flags.go:64] FLAG: --hostname-override="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625935 4981 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625940 4981 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625944 4981 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625951 4981 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 22:23:54.635554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625957 4981 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625961 4981 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625966 4981 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625971 4981 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625975 4981 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625980 4981 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625987 4981 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625991 4981 flags.go:64] FLAG: --kube-reserved="" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.625995 4981 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626000 4981 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626005 4981 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626010 4981 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626014 4981 flags.go:64] FLAG: --lock-file="" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626018 4981 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626022 4981 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626027 4981 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626036 4981 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626040 4981 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626044 4981 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626051 4981 flags.go:64] FLAG: --logging-format="text" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626056 4981 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626061 4981 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626066 4981 flags.go:64] FLAG: --manifest-url="" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626071 4981 flags.go:64] FLAG: --manifest-url-header="" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626077 4981 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 22:23:54.636533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626083 4981 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626089 4981 flags.go:64] FLAG: --max-pods="110" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626094 4981 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626099 4981 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626105 4981 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626110 4981 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626115 4981 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626119 4981 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626124 4981 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626136 4981 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626141 4981 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626146 4981 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626151 4981 flags.go:64] FLAG: --pod-cidr="" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626157 4981 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626162 4981 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626167 4981 flags.go:64] FLAG: --pods-per-core="0" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626171 4981 flags.go:64] FLAG: --port="10250" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626176 4981 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626181 4981 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0dec6a2294b88ef75" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626186 4981 flags.go:64] FLAG: --qos-reserved="" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626191 4981 flags.go:64] FLAG: --read-only-port="10255" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626197 4981 flags.go:64] FLAG: --register-node="true" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626201 4981 flags.go:64] FLAG: --register-with-taints="" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626212 4981 flags.go:64] FLAG: --registry-burst="10" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626217 4981 flags.go:64] FLAG: --registry-qps="5" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626222 4981 flags.go:64] FLAG: --reserved-cpus="" Nov 01 22:23:54.637428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626227 4981 flags.go:64] FLAG: --reserved-memory="" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626233 4981 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626238 4981 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626242 4981 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626247 4981 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626251 4981 flags.go:64] FLAG: --runonce="false" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626256 4981 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626262 4981 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626267 4981 flags.go:64] FLAG: --seccomp-default="false" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626272 4981 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626276 4981 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626282 4981 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626286 4981 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626293 4981 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626297 4981 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626303 4981 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626308 4981 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626314 4981 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626320 4981 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626326 4981 flags.go:64] FLAG: --system-cgroups="" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626331 4981 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626340 4981 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626345 4981 flags.go:64] FLAG: --tls-cert-file="" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626349 4981 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626356 4981 flags.go:64] FLAG: --tls-min-version="" Nov 01 22:23:54.638323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626361 4981 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626366 4981 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626370 4981 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626375 4981 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626380 4981 flags.go:64] FLAG: --v="2" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626386 4981 flags.go:64] FLAG: --version="false" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626391 4981 flags.go:64] FLAG: --vmodule="" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626397 4981 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.626402 4981 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626608 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626617 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626623 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626627 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626632 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626635 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626639 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626643 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626649 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626655 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626661 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626666 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.639238 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626671 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626675 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626680 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626685 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626690 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626697 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626702 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626706 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626711 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626713 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626716 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626719 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626722 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626724 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626727 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626729 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626732 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626735 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626739 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626743 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.639862 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626748 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626753 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626757 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626761 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626763 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626766 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626769 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626772 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626775 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626777 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626780 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626782 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626785 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626788 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626790 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626793 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626795 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626799 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626802 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.640482 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626804 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626807 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626810 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626812 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626815 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626817 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626820 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626823 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626825 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626828 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626831 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626833 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626836 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626840 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626844 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626847 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626849 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626852 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626855 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.641152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626858 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626861 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626863 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626866 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626869 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626871 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626874 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626876 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626879 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626882 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626885 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626889 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626891 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626894 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626896 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626900 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626904 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626908 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626913 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626916 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.641911 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626921 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626926 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626931 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626935 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626939 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626942 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626944 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626947 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626950 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626952 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626954 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626957 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626959 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626962 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626964 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.626967 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.642747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.627004 4981 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627040 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627044 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627047 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627050 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627053 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627058 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627062 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627065 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627067 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627070 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627073 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627075 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627078 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627081 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627083 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627085 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627088 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627091 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627093 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.643321 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627096 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627098 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627101 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627103 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627106 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627108 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627112 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627116 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627120 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627124 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627129 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627132 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627136 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627139 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627141 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627143 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627146 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627148 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627152 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627157 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.643981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627159 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627162 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627164 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627167 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627170 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627172 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627175 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627177 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627181 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627185 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627188 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627190 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627193 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627195 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627198 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627201 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627203 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627206 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627208 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.644651 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627211 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627213 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627216 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627218 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627221 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627223 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627227 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627229 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627233 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627238 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627242 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627248 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627254 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627259 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627264 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627268 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627272 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627277 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627282 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627288 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.645230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627292 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627295 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627298 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627300 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627303 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627306 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627310 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627315 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627319 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627324 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627329 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627334 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627338 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627342 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627347 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627350 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627353 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627356 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627359 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.645863 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627364 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627369 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627374 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627378 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627383 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627388 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627395 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627403 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.627410 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.635315 4981 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.635336 4981 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.635360 4981 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.635367 4981 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635448 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635456 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635461 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635466 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635470 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635474 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.646431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635478 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635483 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635487 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635491 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635494 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635499 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635503 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635507 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635511 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635516 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635520 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635524 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635527 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635532 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635536 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635540 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635544 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635550 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635555 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635559 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.647088 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635563 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635567 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635704 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635710 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635715 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635719 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635724 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635728 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635732 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635736 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635740 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635744 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635747 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635750 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635755 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635761 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635766 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635772 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635777 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.647614 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635782 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635797 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635802 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635807 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635811 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635815 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635819 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635823 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635827 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635832 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635836 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635841 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635845 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635849 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635853 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635857 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635861 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635865 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635870 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635875 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.648105 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635879 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635883 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635888 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635892 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635896 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635900 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635904 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635909 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635913 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635917 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635921 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635926 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635931 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635935 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635947 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635952 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635956 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635960 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635964 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.648591 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635969 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635975 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635984 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635989 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635994 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.635999 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636004 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636008 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636014 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636018 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636022 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636027 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636031 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636036 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636040 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636044 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636048 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636052 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636057 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.649153 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636062 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636066 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636071 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.636113 4981 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636316 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636324 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636329 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636334 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636339 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636352 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636357 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636361 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636366 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636370 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636376 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636381 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.649736 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636386 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636390 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636396 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636401 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636405 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636408 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636413 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636417 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636421 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636425 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636430 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636434 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636438 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636442 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636446 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636451 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636455 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636459 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636463 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636468 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.650273 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636472 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636476 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636480 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636483 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636486 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636488 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636494 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636496 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636499 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636501 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636504 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636507 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636510 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636514 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636518 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636525 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636532 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636538 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636544 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.651121 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636548 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636553 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636559 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636564 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636569 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636588 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636593 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636597 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636600 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636603 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636606 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636608 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636611 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636614 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636617 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636620 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636624 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636628 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636632 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636638 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.651705 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636645 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636650 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636654 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636658 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636662 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636666 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636670 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636676 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636680 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636685 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636689 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636693 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636697 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636701 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636705 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636709 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636713 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636718 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636722 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.652368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636727 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636732 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636737 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636741 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636745 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636748 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636750 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636753 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636756 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636760 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636764 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636768 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636772 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636775 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636779 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636781 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.652955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.636826 4981 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636874 4981 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636881 4981 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636886 4981 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636891 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636895 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636899 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636903 4981 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636907 4981 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636911 4981 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636915 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636920 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636924 4981 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636928 4981 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636932 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636936 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636940 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636945 4981 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636950 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 22:23:54.653456 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636954 4981 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636959 4981 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636964 4981 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636967 4981 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636970 4981 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636973 4981 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636976 4981 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636981 4981 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636985 4981 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636990 4981 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636994 4981 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.636998 4981 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637002 4981 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637004 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637007 4981 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637010 4981 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637012 4981 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637014 4981 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637017 4981 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637019 4981 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 22:23:54.654214 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637022 4981 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637024 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637027 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637029 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637032 4981 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637035 4981 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637038 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637041 4981 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637047 4981 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637055 4981 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637060 4981 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637065 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637070 4981 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637075 4981 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637080 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637084 4981 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637094 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637099 4981 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637104 4981 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 22:23:54.654998 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637108 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637113 4981 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637117 4981 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637120 4981 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637123 4981 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637127 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637132 4981 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637137 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637142 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637146 4981 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637150 4981 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637154 4981 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637159 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637164 4981 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637169 4981 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637174 4981 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637179 4981 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637185 4981 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637188 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637191 4981 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 22:23:54.655506 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637193 4981 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637196 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637198 4981 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637201 4981 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637204 4981 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637207 4981 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637209 4981 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637212 4981 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637214 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637218 4981 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637220 4981 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637223 4981 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637225 4981 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637228 4981 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637230 4981 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637233 4981 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637235 4981 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637240 4981 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637242 4981 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 22:23:54.656248 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637246 4981 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637249 4981 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637251 4981 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637254 4981 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637256 4981 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637260 4981 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637264 4981 feature_gate.go:386] unrecognized feature gate: Example Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637268 4981 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637273 4981 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:54.637278 4981 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.637363 4981 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.638710 4981 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.638778 4981 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.639030 4981 server.go:1014] "Starting client certificate rotation" Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.639185 4981 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.639249 4981 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:42 +0000 UTC" deadline="2028-06-18 23:01:50.512179283 +0000 UTC" Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.639286 4981 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14280h37m55.872895917s" Nov 01 22:23:54.656874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.639735 4981 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:54.657420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.641322 4981 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 22:23:54.657420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.643040 4981 log.go:25] "Validated CRI v1 runtime API" Nov 01 22:23:54.657420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.645034 4981 log.go:25] "Validated CRI v1 image API" Nov 01 22:23:54.657420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.646044 4981 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 22:23:54.657420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.648088 4981 fs.go:135] Filesystem UUIDs: map[21a393e9-ecd7-498c-aa8c-f2de1e1e757b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d7de22f9-ddf5-4e77-9124-c3a7b309743c:/dev/nvme0n1p4] Nov 01 22:23:54.657622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.648107 4981 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a6fcdb1ebaaaed5ab5e6f1063eff317ff584f1d208faaffea90a551194c85e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a6fcdb1ebaaaed5ab5e6f1063eff317ff584f1d208faaffea90a551194c85e1/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a0364f0eaa5303dc1e100cd28a83623349d7f83c9666c8b73693600baeafd79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a0364f0eaa5303dc1e100cd28a83623349d7f83c9666c8b73693600baeafd79/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7370cc4b591de15db84fba0d4bf74d6fe09ed0143ce859685086c3405df4714b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7370cc4b591de15db84fba0d4bf74d6fe09ed0143ce859685086c3405df4714b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dbb553234b4354d81ac73f479623608c1e4605a15ebca764e4daa16798dae23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dbb553234b4354d81ac73f479623608c1e4605a15ebca764e4daa16798dae23/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8022477da90b6752f2e4d3d8c7c045597dcf42c2cd6a37833f85355db84dc869/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8022477da90b6752f2e4d3d8c7c045597dcf42c2cd6a37833f85355db84dc869/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90332090f685da3f426e853bbc93d6a110a17f8889027bf81fa4d9995dc28dfa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90332090f685da3f426e853bbc93d6a110a17f8889027bf81fa4d9995dc28dfa/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9629a8c8bb23451632cf19c51d7d9fe02cd060bbd708d7918faee8f7659a7222/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9629a8c8bb23451632cf19c51d7d9fe02cd060bbd708d7918faee8f7659a7222/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8332fdaf88ac16b69fab9b38a0b8ced211975b1ae3e93d3142a5ddac2f1de62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8332fdaf88ac16b69fab9b38a0b8ced211975b1ae3e93d3142a5ddac2f1de62/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abefdd856761d8cb798c3204eb491c6d0f3e4a47d038e9823b916b0772ac638e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abefdd856761d8cb798c3204eb491c6d0f3e4a47d038e9823b916b0772ac638e/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd68110155231b38a3b26e68d7fdaa9ae06512c587ba785601601c3d9dfc6d0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd68110155231b38a3b26e68d7fdaa9ae06512c587ba785601601c3d9dfc6d0a/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb86560c6b8a26ab17c9105a237a46b708eaa0e2d4c4de0e35dcfe1babf79868/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb86560c6b8a26ab17c9105a237a46b708eaa0e2d4c4de0e35dcfe1babf79868/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf55523618275778e7c088a4918a47a403f3dd406d6361c5f4bf37e538dd816b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf55523618275778e7c088a4918a47a403f3dd406d6361c5f4bf37e538dd816b/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4c9d31ef980c288441b066a4dc7a8a92a6e682b6c8ee49bc8ae4a05e329ff89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4c9d31ef980c288441b066a4dc7a8a92a6e682b6c8ee49bc8ae4a05e329ff89/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec988848c8755e82f264fb0d5d373d798325e9d63842cc35b8054655a4664ca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec988848c8755e82f264fb0d5d373d798325e9d63842cc35b8054655a4664ca7/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f09a1075b225f6706e04a61bb03794e5a73bd0a3ce2490386d2d4d11e660539f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f09a1075b225f6706e04a61bb03794e5a73bd0a3ce2490386d2d4d11e660539f/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f73965ed56ae4f813336ce927838ac356df9e7144711904f5a38304c1842f781/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f73965ed56ae4f813336ce927838ac356df9e7144711904f5a38304c1842f781/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa635b1ed172ae10c482eee2fe146282932248b864e9416a24975283e3c54ea3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa635b1ed172ae10c482eee2fe146282932248b864e9416a24975283e3c54ea3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f3425f3-64ee-4134-b096-10e48c0c5faa/volumes/kubernetes.io~projected/kube-api-access-lcpmt:{mountpoint:/var/lib/kubelet/pods/0f3425f3-64ee-4134-b096-10e48c0c5faa/volumes/kubernetes.io~projected/kube-api-access-lcpmt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~projected/kube-api-access-l6gnk:{mountpoint:/var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~projected/kube-api-access-l6gnk major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858/volumes/kubernetes.io~projected/kube-api-access-wwz2q:{mountpoint:/var/lib/kubelet/pods/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858/volumes/kubernetes.io~projected/kube-api-access-wwz2q major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47eb4322-3268-4e50-80e0-35f2dd802b0b/volumes/kubernetes.io~projected/kube-api-access-qrbtt:{mountpoint:/var/lib/kubelet/pods/47eb4322-3268-4e50-80e0-35f2dd802b0b/volumes/kubernetes.io~projected/kube-api-access-qrbtt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47eb4322-3268-4e50-80e0-35f2dd802b0b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/47eb4322-3268-4e50-80e0-35f2dd802b0b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e15bc3a-dc26-4b68-bba9-b670d741c952/volumes/kubernetes.io~projected/kube-api-access-z46gn:{mountpoint:/var/lib/kubelet/pods/4e15bc3a-dc26-4b68-bba9-b670d741c952/volumes/kubernetes.io~projected/kube-api-access-z46gn major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e15bc3a-dc26-4b68-bba9-b670d741c952/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4e15bc3a-dc26-4b68-bba9-b670d741c952/volumes/kubernetes.io~secret/cert major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~projected/kube-api-access-56zkk:{mountpoint:/var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~projected/kube-api-access-56zkk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c512618-f7bd-4210-90f9-e525e4a8a1b7/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/5c512618-f7bd-4210-90f9-e525e4a8a1b7/volumes/kubernetes.io~secret/tls-certificates major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7025fde7-1830-4d2e-9780-a4c0a8195d0f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7025fde7-1830-4d2e-9780-a4c0a8195d0f/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7caa0db8-4f8f-4848-a2a6-845f20c9a888/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7caa0db8-4f8f-4848-a2a6-845f20c9a888/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7eeac783-1128-4402-8404-5962939c1b63/volumes/kubernetes.io~projected/kube-api-access-82s88:{mountpoint:/var/lib/kubelet/pods/7eeac783-1128-4402-8404-5962939c1b63/volumes/kubernetes.io~projected/kube-api-access-82s88 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/kube-api-access-km45f:{mountpoint:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/kube-api-access-km45f major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/registry-tls major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/874207d9-9be6-4544-bc12-2bf935d7e477/volumes/kubernetes.io~projected/kube-api-access-2w2w7:{mountpoint:/var/lib/kubelet/pods/874207d9-9be6-4544-bc12-2bf935d7e477/volumes/kubernetes.io~projected/kube-api-access-2w2w7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/874207d9-9be6-4544-bc12-2bf935d7e477/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/874207d9-9be6-4544-bc12-2bf935d7e477/volumes/kubernetes.io~secret/metrics-tls major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8deca68d-1a0b-4841-9c91-0682262d59c7/volumes/kubernetes.io~projected/kube-api-access-s28ks:{mountpoint:/var/lib/kubelet/pods/8deca68d-1a0b-4841-9c91-0682262d59c7/volumes/kubernetes.io~projected/kube-api-access-s28ks major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/948069e6-a5b8-407d-a12b-198a76a2e783/volumes/kubernetes.io~projected/kube-api-access-55hgz:{mountpoint:/var/lib/kubelet/pods/948069e6-a5b8-407d-a12b-198a76a2e783/volumes/kubernetes.io~projected/kube-api-access-55hgz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4ce9767-19f2-42a8-afe1-ad361726b8b9/volumes/kubernetes.io~projected/kube-api-access-xk5z4:{mountpoint:/var/lib/kubelet/pods/a4ce9767-19f2-42a8-afe1-ad361726b8b9/volumes/kubernetes.io~projected/kube-api-access-xk5z4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa89357c-1284-4f7a-b319-fcdb18edb560/volumes/kubernetes.io~projected/kube-api-access-887gs:{mountpoint:/var/lib/kubelet/pods/aa89357c-1284-4f7a-b319-fcdb18edb560/volumes/kubernetes.io~projected/kube-api-access-887gs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9cf0fb4-ef5d-4921-9c01-8830c386dce8/volumes/kubernetes.io~projected/kube-api-access-dm8f9:{mountpoint:/var/lib/kubelet/pods/d9cf0fb4-ef5d-4921-9c01-8830c386dce8/volumes/kubernetes.io~projected/kube-api-access-dm8f9 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9cf0fb4-ef5d-4921-9c01-8830c386dce8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d9cf0fb4-ef5d-4921-9c01-8830c386dce8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volumes/kubernetes.io~projected/kube-api-access-fdbpk:{mountpoint:/var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volumes/kubernetes.io~projected/kube-api-access-fdbpk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f130f8c5-0161-4ef2-8634-cc150d09ddf0/volumes/kubernetes.io~projected/kube-api-access-cl5b8:{mountpoint:/var/lib/kubelet/pods/f130f8c5-0161-4ef2-8634-cc150d09ddf0/volumes/kubernetes.io~projected/kube-api-access-cl5b8 major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/dfe1b0a608920fc57d496b1f96a46d786473bd18cd4dceee382d032f4283db2c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/bc4b3b8f041bde6c6b8d09bb2b2b610fe37ee570e00d86d908158aa4c138a013/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f2e24724b7fd4f899f1beb9331ad20edc63dadcd9ead6e405832a1ff09d5d1cf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/75ded2e6b12dbcd70fb23d47a6fc4e8d8509df5c6cf3cda3cc5d53190a1023da/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/d9ef38039a114f9d331f46dac7c8e119e4a2f3a6f9e5a9f86f1d12af2e689a77/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7330ed02b05b54d9e81a80022618d3871f44e7795ecba3c061a4c20d039e5e31/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8d2971cb5f54b99f6303a5a85c8563fec0737954dde89779432c8a9b36cdd372/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8580046aebc01b9e878f2b3ae11127f39900f2b197eb8ec10fae4b0098f9a524/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/faf80050dcbd4f3d9311656eb255a265b4fb4fb7f06625ce9161ea12a15f40c3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/cb9eede3b4b130fdf5fd6762b9e14b94ecff7ac32648241933d11b8689ea3273/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f1c09cc089af8c881b74d364e6afb55ff12a7801986eabdec857e3d531cfe7c5/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/9bb6c5f06808acc331e01234541edb2c8331d7d4055c97591a5aae3b1d0949b7/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d74de640431eb1c58c0d8cd13085cee084fad8c0f60f74490382a45add0e1e0c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9fb8506909f7b10b799842843a3c93397f037c81c9fe70f1cec0e42c035bd98d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/11eec52f1673f2ede34ac3d4cdf42316b5e4da4ffeb6f090596d4266d27e4a81/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/385619189e1197901eaaeffce742f2b01d2cf9287f6e79d3eb209b8e86a92b61/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/398cc4dde686f772808a2a01999a36460209b690f66dfd12d82ddd2b6d3bd8ee/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/520551152bb23afc3d3b05a4c7c3df060e41ade6098a42266314a33f859b4f60/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3fddc09370ced613b5c91430b67e22220a6c1d67ef170f53cf69d9db485a6cf8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a31a33d4892f47fd8b51e329da5977aafe28d530706b3c740a6a75f7921246e5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/2c5cbbe3b3cfcc3405d29f9c8c15f6371c86411ef8381385a5c8280041fd8f0a/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/4b27b1223008a9331ac9d849d8bb4a11755ca0056e864ceb33160c12917a7ed4/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/7f705735f645849078a0f0c382991f4534d24262ecc88e95b6efb0806c13fb01/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/b3d08b34e7ba4ec266348181ebb88b99e472ae0befb8b3c37fd133427d112a6c/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/2e467eea138656ec53a9a682251481b518c292fd07addcf7711aa68f947d69c6/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/914e525bc11020256a8e1eeab8e31007852c152a35d6d546d9ed2427ea553101/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/e42e245636652da3eb02432cfca0d3f540b84eef13f889764416b95270a0094d/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/9b2aac267420ec7e9a2962e8a4aac365a3728b08f8cab628ae31a822cee03ac4/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/05bb51244596eb21dbc55dffabe8dfc8a01d0b38558a46ce3e63cfcfb34a7cdc/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/46de03cecb569657a1d3dc3680024d07bac86907b1c8fe04aad43477f91d39a8/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d9cb34318b0126ff9fef35752dc3d82e5a0dce21a297ab00ad6677e6bed3f7ce/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/99e74252cc62a868f5ad55ab0500dc394a07d1796eb279fa7c73729145b67112/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/1f10a3496fe606d5547a7b8824ac28e8b567cbab5c99ba54a30c3f7729af5f8a/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/392a95bba65d34cf8d4847db5cc5dd9bda82f35482d39001c8900068cb4b876f/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/fa05e95521264fdf87f87d69b8a50770c0daee55552fd2fedbc8385846ba73fd/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/f9a264cbde002ad9da0257530c67e7d0fa5b19a0d8daf832e469eb8b5683c099/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/4ce0f1e50de1f40daee6cc9e61311185bbd7595fd8416d5cdb32dc63288c551e/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/1ca8e5d1c34c0f2faf4dd16ffda8f53b2b5a8423c23f729dceb22bae34db31d5/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/00ad89a2e92c65b7aab82e697196ec7ee4f950eefc9911e469b5c62b766b688c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/9b2a74fa1cb457deefaa8fd4603d5fd172ca34e1cb3cc15fb45926f4683aaf01/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/04e490893764b485545fb57ad98591beff5fcd7d314055cd09eccf7a561ebc4d/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/a58f03445f02ca209ac6275b2e94f0d6eb99927a251303632ce8070820e73a6a/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/31c56a130f59f07543e894a57fd07455b0f5860f12b120d9d3530145c9765fe4/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a2672ef1d73d024531bce8652d4a0670b739112183f15bfa6e256446c65e3c38/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f7a2c398f3ee7ee94a8dc6477b335642c29664557bcdead51a3676503f8004b3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f7b59c960d3b7cfaaee10752052daaaa547e8740a01a82ba41a9c8f9c2897740/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b8f08e827445597c0b98c0a1be3276b4c7f48d4ec4d26812ff570fa63baa9241/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3f13f1d96f68c52e34a39ca8718512b1643b0e92bf02b33944a434b580840ca4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2e20385661fb752d73aa76fd0ef2fd9cd79004a452b0ba234afb4fad93f65444/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4d7372fbd6ef23b4ac029f4c51b11fe63a6d962b877492fb7d7a3a376a109bb5/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/eefcc5c799b40fd7fdc81a0abdf5f05a23e10356711f9f5930e073272fd7a4b6/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/83cc5453ac3dd255e09991be316fe91a18c66dbc626942f400ef37c487e275d4/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 01 22:23:54.658409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.657861 4981 manager.go:217] Machine: {Timestamp:2026-11-01 22:23:54.65701476 +0000 UTC m=+0.084892684 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3202372 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b7b716830ad78fe5215868d76f714 SystemUUID:ec2b7b71-6830-ad78-fe52-15868d76f714 BootID:f43fc64b-0361-4ff0-bb62-e39f87d63435 Filesystems:[{Device:/run/containers/storage/overlay-containers/2a0364f0eaa5303dc1e100cd28a83623349d7f83c9666c8b73693600baeafd79/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c512618-f7bd-4210-90f9-e525e4a8a1b7/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:267 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf55523618275778e7c088a4918a47a403f3dd406d6361c5f4bf37e538dd816b/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8deca68d-1a0b-4841-9c91-0682262d59c7/volumes/kubernetes.io~projected/kube-api-access-s28ks DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4c9d31ef980c288441b066a4dc7a8a92a6e682b6c8ee49bc8ae4a05e329ff89/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f09a1075b225f6706e04a61bb03794e5a73bd0a3ce2490386d2d4d11e660539f/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/874207d9-9be6-4544-bc12-2bf935d7e477/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:196 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9cf0fb4-ef5d-4921-9c01-8830c386dce8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47eb4322-3268-4e50-80e0-35f2dd802b0b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7caa0db8-4f8f-4848-a2a6-845f20c9a888/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90332090f685da3f426e853bbc93d6a110a17f8889027bf81fa4d9995dc28dfa/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dbb553234b4354d81ac73f479623608c1e4605a15ebca764e4daa16798dae23/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa89357c-1284-4f7a-b319-fcdb18edb560/volumes/kubernetes.io~projected/kube-api-access-887gs DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb86560c6b8a26ab17c9105a237a46b708eaa0e2d4c4de0e35dcfe1babf79868/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/874207d9-9be6-4544-bc12-2bf935d7e477/volumes/kubernetes.io~projected/kube-api-access-2w2w7 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9629a8c8bb23451632cf19c51d7d9fe02cd060bbd708d7918faee8f7659a7222/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec988848c8755e82f264fb0d5d373d798325e9d63842cc35b8054655a4664ca7/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~projected/kube-api-access-l6gnk DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7025fde7-1830-4d2e-9780-a4c0a8195d0f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4ce9767-19f2-42a8-afe1-ad361726b8b9/volumes/kubernetes.io~projected/kube-api-access-xk5z4 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa635b1ed172ae10c482eee2fe146282932248b864e9416a24975283e3c54ea3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7370cc4b591de15db84fba0d4bf74d6fe09ed0143ce859685086c3405df4714b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4e15bc3a-dc26-4b68-bba9-b670d741c952/volumes/kubernetes.io~projected/kube-api-access-z46gn DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a6fcdb1ebaaaed5ab5e6f1063eff317ff584f1d208faaffea90a551194c85e1/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd68110155231b38a3b26e68d7fdaa9ae06512c587ba785601601c3d9dfc6d0a/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/948069e6-a5b8-407d-a12b-198a76a2e783/volumes/kubernetes.io~projected/kube-api-access-55hgz DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f130f8c5-0161-4ef2-8634-cc150d09ddf0/volumes/kubernetes.io~projected/kube-api-access-cl5b8 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8332fdaf88ac16b69fab9b38a0b8ced211975b1ae3e93d3142a5ddac2f1de62/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8022477da90b6752f2e4d3d8c7c045597dcf42c2cd6a37833f85355db84dc869/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858/volumes/kubernetes.io~projected/kube-api-access-wwz2q DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f73965ed56ae4f813336ce927838ac356df9e7144711904f5a38304c1842f781/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e91e58ba-dbc6-49da-b601-22edc7b6525e/volumes/kubernetes.io~projected/kube-api-access-fdbpk DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e15bc3a-dc26-4b68-bba9-b670d741c952/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:200 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abefdd856761d8cb798c3204eb491c6d0f3e4a47d038e9823b916b0772ac638e/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7eeac783-1128-4402-8404-5962939c1b63/volumes/kubernetes.io~projected/kube-api-access-82s88 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/47eb4322-3268-4e50-80e0-35f2dd802b0b/volumes/kubernetes.io~projected/kube-api-access-qrbtt DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0f3425f3-64ee-4134-b096-10e48c0c5faa/volumes/kubernetes.io~projected/kube-api-access-lcpmt DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83ec6a1b-db92-4103-adec-8539c8bb86c2/volumes/kubernetes.io~projected/kube-api-access-km45f DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d9cf0fb4-ef5d-4921-9c01-8830c386dce8/volumes/kubernetes.io~projected/kube-api-access-dm8f9 DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/548c3a65-3217-4609-b266-b7eacca269a3/volumes/kubernetes.io~projected/kube-api-access-56zkk DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a6fcdb1ebaaaed MacAddress:96:04:60:12:bb:5a Speed:10000 Mtu:8901} {Name:2a0364f0eaa5303 MacAddress:62:c0:26:27:41:db Speed:10000 Mtu:8901} {Name:48fa1eb121bf952 MacAddress:da:66:7e:e4:1c:fb Speed:10000 Mtu:8901} {Name:8022477da90b675 MacAddress:52:24:e7:69:aa:31 Speed:10000 Mtu:8901} {Name:9629a8c8bb23451 MacAddress:7e:32:8c:7e:62:40 Speed:10000 Mtu:8901} {Name:abefdd856761d8c MacAddress:3e:af:3b:ff:88:77 Speed:10000 Mtu:8901} {Name:bd68110155231b3 MacAddress:72:14:1a:82:cb:46 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e2:87:3c:6d Speed:0 Mtu:9001} {Name:br-int MacAddress:9a:a6:d3:2e:b9:d8 Speed:0 Mtu:8901} {Name:e4c9d31ef980c28 MacAddress:9e:53:57:c4:86:35 Speed:10000 Mtu:8901} {Name:ec988848c8755e8 MacAddress:d2:f9:c8:26:37:43 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e2:87:3c:6d Speed:0 Mtu:9001} {Name:f09a1075b225f67 MacAddress:e2:e0:20:cc:f9:2e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ba:0d:89:35:82:90 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:32:24:2b:10:95:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 22:23:54.658409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.658413 4981 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 22:23:54.658853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.658470 4981 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 22:23:54.658853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.658799 4981 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 22:23:54.659003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.658820 4981 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-87.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 22:23:54.659003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659004 4981 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 22:23:54.659122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659016 4981 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 22:23:54.659122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659037 4981 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:54.659122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659063 4981 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 22:23:54.659237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659163 4981 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 22:23:54.659363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659351 4981 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 22:23:54.659513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659502 4981 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 22:23:54.659638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659544 4981 kubelet.go:493] "Attempting to sync node with API server" Nov 01 22:23:54.659638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659558 4981 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 22:23:54.659638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659570 4981 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 22:23:54.659638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659596 4981 kubelet.go:405] "Adding apiserver pod source" Nov 01 22:23:54.659638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.659608 4981 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 22:23:54.660537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.660511 4981 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:54.660537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.660528 4981 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:54.661083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.661054 4981 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.673728 4981 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.673745 4981 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.673880 4981 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674067 4981 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674096 4981 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674195 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674207 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674214 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674219 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674225 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674232 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674241 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674249 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674256 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674262 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674276 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674290 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674320 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.674329 4981 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 22:23:54.676600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.675326 4981 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:54.678098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.678077 4981 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:54.678393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.678381 4981 server.go:1263] "Started kubelet" Nov 01 22:23:54.679198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.679113 4981 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 22:23:54.679165 ip-10-0-138-87 systemd[1]: Started Kubernetes Kubelet. Nov 01 22:23:54.680750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.679945 4981 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 22:23:54.680750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.680085 4981 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 22:23:54.682258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.681446 4981 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 22:23:54.682258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.681558 4981 server.go:317] "Adding debug handlers to kubelet server" Nov 01 22:23:54.687698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.687381 4981 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 22:23:54.687698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.687422 4981 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 22:23:54.687698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.687487 4981 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 22:23:54.687698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.687496 4981 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 22:23:54.687698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.687499 4981 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:44 +0000 UTC" deadline="2028-05-05 13:53:36.680728562 +0000 UTC" Nov 01 22:23:54.687698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.687530 4981 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13215h29m41.993201216s" Nov 01 22:23:54.687698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.687570 4981 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689201 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cf0fb4-ef5d-4921-9c01-8830c386dce8" volumeName="kubernetes.io/projected/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-api-access-dm8f9" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689242 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe7ab41-3bca-400b-9a55-ed06fb29c216" volumeName="kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689259 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-tls" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689272 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-bound-sa-token" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689285 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa89357c-1284-4f7a-b319-fcdb18edb560" volumeName="kubernetes.io/projected/aa89357c-1284-4f7a-b319-fcdb18edb560-kube-api-access-887gs" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689296 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e91e58ba-dbc6-49da-b601-22edc7b6525e" volumeName="kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-env-overrides" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689311 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f130f8c5-0161-4ef2-8634-cc150d09ddf0" volumeName="kubernetes.io/configmap/f130f8c5-0161-4ef2-8634-cc150d09ddf0-serviceca" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689322 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe7ab41-3bca-400b-9a55-ed06fb29c216" volumeName="kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689337 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe7ab41-3bca-400b-9a55-ed06fb29c216" volumeName="kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689347 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47eb4322-3268-4e50-80e0-35f2dd802b0b" volumeName="kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689359 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-installation-pull-secrets" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689370 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="948069e6-a5b8-407d-a12b-198a76a2e783" volumeName="kubernetes.io/projected/948069e6-a5b8-407d-a12b-198a76a2e783-kube-api-access-55hgz" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689382 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4ce9767-19f2-42a8-afe1-ad361726b8b9" volumeName="kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689394 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa89357c-1284-4f7a-b319-fcdb18edb560" volumeName="kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-daemon-config" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689408 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e91e58ba-dbc6-49da-b601-22edc7b6525e" volumeName="kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-config" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689419 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe7ab41-3bca-400b-9a55-ed06fb29c216" volumeName="kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689430 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4713d5-cc55-4d7e-86f8-f1e17f3b3858" volumeName="kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689441 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e15bc3a-dc26-4b68-bba9-b670d741c952" volumeName="kubernetes.io/secret/4e15bc3a-dc26-4b68-bba9-b670d741c952-cert" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689453 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeac783-1128-4402-8404-5962939c1b63" volumeName="kubernetes.io/configmap/7eeac783-1128-4402-8404-5962939c1b63-iptables-alerter-script" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689465 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cf0fb4-ef5d-4921-9c01-8830c386dce8" volumeName="kubernetes.io/configmap/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689476 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e91e58ba-dbc6-49da-b601-22edc7b6525e" volumeName="kubernetes.io/projected/e91e58ba-dbc6-49da-b601-22edc7b6525e-kube-api-access-fdbpk" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689488 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f3425f3-64ee-4134-b096-10e48c0c5faa" volumeName="kubernetes.io/projected/0f3425f3-64ee-4134-b096-10e48c0c5faa-kube-api-access-lcpmt" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689499 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548c3a65-3217-4609-b266-b7eacca269a3" volumeName="kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-tmp" seLinuxMountContext="" Nov 01 22:23:54.689498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689510 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7eeac783-1128-4402-8404-5962939c1b63" volumeName="kubernetes.io/projected/7eeac783-1128-4402-8404-5962939c1b63-kube-api-access-82s88" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689524 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874207d9-9be6-4544-bc12-2bf935d7e477" volumeName="kubernetes.io/empty-dir/874207d9-9be6-4544-bc12-2bf935d7e477-tmp-dir" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689537 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8deca68d-1a0b-4841-9c91-0682262d59c7" volumeName="kubernetes.io/empty-dir/8deca68d-1a0b-4841-9c91-0682262d59c7-tmp" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689548 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe7ab41-3bca-400b-9a55-ed06fb29c216" volumeName="kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689560 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe7ab41-3bca-400b-9a55-ed06fb29c216" volumeName="kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689606 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-certificates" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689618 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874207d9-9be6-4544-bc12-2bf935d7e477" volumeName="kubernetes.io/configmap/874207d9-9be6-4544-bc12-2bf935d7e477-config-volume" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689630 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874207d9-9be6-4544-bc12-2bf935d7e477" volumeName="kubernetes.io/secret/874207d9-9be6-4544-bc12-2bf935d7e477-metrics-tls" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689653 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e91e58ba-dbc6-49da-b601-22edc7b6525e" volumeName="kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-script-lib" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689664 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f130f8c5-0161-4ef2-8634-cc150d09ddf0" volumeName="kubernetes.io/projected/f130f8c5-0161-4ef2-8634-cc150d09ddf0-kube-api-access-cl5b8" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689675 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fe7ab41-3bca-400b-9a55-ed06fb29c216" volumeName="kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689686 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548c3a65-3217-4609-b266-b7eacca269a3" volumeName="kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-etc-tuned" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689697 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="548c3a65-3217-4609-b266-b7eacca269a3" volumeName="kubernetes.io/projected/548c3a65-3217-4609-b266-b7eacca269a3-kube-api-access-56zkk" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689708 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7025fde7-1830-4d2e-9780-a4c0a8195d0f" volumeName="kubernetes.io/configmap/7025fde7-1830-4d2e-9780-a4c0a8195d0f-konnectivity-ca" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689719 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/empty-dir/83ec6a1b-db92-4103-adec-8539c8bb86c2-ca-trust-extracted" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689731 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-kube-api-access-km45f" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689742 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8deca68d-1a0b-4841-9c91-0682262d59c7" volumeName="kubernetes.io/projected/8deca68d-1a0b-4841-9c91-0682262d59c7-kube-api-access-s28ks" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689752 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="948069e6-a5b8-407d-a12b-198a76a2e783" volumeName="kubernetes.io/empty-dir/948069e6-a5b8-407d-a12b-198a76a2e783-tmp-dir" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689765 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47eb4322-3268-4e50-80e0-35f2dd802b0b" volumeName="kubernetes.io/projected/47eb4322-3268-4e50-80e0-35f2dd802b0b-kube-api-access-qrbtt" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689787 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7025fde7-1830-4d2e-9780-a4c0a8195d0f" volumeName="kubernetes.io/secret/7025fde7-1830-4d2e-9780-a4c0a8195d0f-agent-certs" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689804 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="874207d9-9be6-4544-bc12-2bf935d7e477" volumeName="kubernetes.io/projected/874207d9-9be6-4544-bc12-2bf935d7e477-kube-api-access-2w2w7" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689821 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4ce9767-19f2-42a8-afe1-ad361726b8b9" volumeName="kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689833 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4ce9767-19f2-42a8-afe1-ad361726b8b9" volumeName="kubernetes.io/projected/a4ce9767-19f2-42a8-afe1-ad361726b8b9-kube-api-access-xk5z4" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689844 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa89357c-1284-4f7a-b319-fcdb18edb560" volumeName="kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-cni-binary-copy" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689856 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cf0fb4-ef5d-4921-9c01-8830c386dce8" volumeName="kubernetes.io/secret/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689869 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e91e58ba-dbc6-49da-b601-22edc7b6525e" volumeName="kubernetes.io/secret/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689881 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e15bc3a-dc26-4b68-bba9-b670d741c952" volumeName="kubernetes.io/projected/4e15bc3a-dc26-4b68-bba9-b670d741c952-kube-api-access-z46gn" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689893 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c512618-f7bd-4210-90f9-e525e4a8a1b7" volumeName="kubernetes.io/secret/5c512618-f7bd-4210-90f9-e525e4a8a1b7-tls-certificates" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689914 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7caa0db8-4f8f-4848-a2a6-845f20c9a888" volumeName="kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689925 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-trusted-ca" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689933 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83ec6a1b-db92-4103-adec-8539c8bb86c2" volumeName="kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-image-registry-private-configuration" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689943 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4ce9767-19f2-42a8-afe1-ad361726b8b9" volumeName="kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689951 4981 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9cf0fb4-ef5d-4921-9c01-8830c386dce8" volumeName="kubernetes.io/empty-dir/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-data-volume" seLinuxMountContext="" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689958 4981 reconstruct.go:97] "Volume reconstruction finished" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.689965 4981 reconciler.go:29] "Reconciler: start to sync state" Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690247 4981 factory.go:55] Registering systemd factory Nov 01 22:23:54.690507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690260 4981 factory.go:223] Registration of the systemd container factory successfully Nov 01 22:23:54.691912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690569 4981 factory.go:153] Registering CRI-O factory Nov 01 22:23:54.691912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690599 4981 factory.go:223] Registration of the crio container factory successfully Nov 01 22:23:54.691912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690659 4981 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 22:23:54.691912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690678 4981 factory.go:103] Registering Raw factory Nov 01 22:23:54.691912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690693 4981 manager.go:1199] Started watching for new ooms in manager Nov 01 22:23:54.691912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.690949 4981 manager.go:319] Starting recovery of all containers Nov 01 22:23:54.693419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.693404 4981 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:54.693633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.693619 4981 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 22:23:54.723164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.723132 4981 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 22:23:54.725634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.725615 4981 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 22:23:54.725799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.725778 4981 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 22:23:54.725908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.725891 4981 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 22:23:54.726016 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:54.725940 4981 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 22:23:54.733053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.732798 4981 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 22:23:54.747721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.747648 4981 generic.go:356] "Generic (PLEG): container finished" podID="bda61bce4694bad4d0a8e6759feac37d" containerID="bc19ee39eaf2d65b3712dbd0f6e8285ed9cb9a2a2ec2297b6889f61075b30d67" exitCode=0 Nov 01 22:23:54.755148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.755117 4981 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="bee4432288650de90c638be26dcc614d8d0d8dbbce6c9736319cb9a9f50867eb" exitCode=0 Nov 01 22:23:54.755251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.755158 4981 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="bde6f6c58862ccddfa92735918e75429cf7f99905db3ceb09597442ffcc7c497" exitCode=0 Nov 01 22:23:54.755251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.755169 4981 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="5d62e6b7ecb22c9e9a9a2a1154403d70ac1c5bb565361f026153a74ff072f4f2" exitCode=0 Nov 01 22:23:54.755251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.755176 4981 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="4ce0763e44b7c1c1cb19e8cb9c24e79b422df507c449274f6033c4f1ed3d51a7" exitCode=0 Nov 01 22:23:54.755251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.755183 4981 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="7ad1d40b3e8b2dd6aa15cf6cbd13eab1a001acb962669d367c935649d3005186" exitCode=0 Nov 01 22:23:54.755251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.755192 4981 generic.go:356] "Generic (PLEG): container finished" podID="a4ce9767-19f2-42a8-afe1-ad361726b8b9" containerID="cf314368adfae09b3102f588be31f648f6316bda6d9a3051783e4222d8a1d921" exitCode=0 Nov 01 22:23:54.786388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.786365 4981 manager.go:324] Recovery completed Nov 01 22:23:54.810659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.810632 4981 cpu_manager.go:226] "Starting" policy="none" Nov 01 22:23:54.810659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.810650 4981 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 22:23:54.810872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.810672 4981 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 22:23:54.810872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.810811 4981 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 22:23:54.811375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.810825 4981 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 22:23:54.811473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811380 4981 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 22:23:54.811473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811389 4981 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 22:23:54.811473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811398 4981 policy_none.go:50] "Start" Nov 01 22:23:54.811473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811409 4981 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 22:23:54.811473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811424 4981 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 22:23:54.811710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811628 4981 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 22:23:54.811710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811639 4981 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 22:23:54.811710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.811648 4981 policy_none.go:44] "Start" Nov 01 22:23:54.817455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.817416 4981 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 22:23:54.820710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.820693 4981 manager.go:341] "Starting Device Plugin manager" Nov 01 22:23:54.820805 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:54.820724 4981 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 22:23:54.820805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.820735 4981 server.go:87] "Starting device plugin registration server" Nov 01 22:23:54.821059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.821043 4981 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 22:23:54.821128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.821062 4981 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 22:23:54.821176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.821154 4981 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 22:23:54.821284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.821248 4981 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 22:23:54.821352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.821297 4981 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 22:23:54.821352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.821305 4981 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 22:23:54.821446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.821365 4981 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:54.821279857Z","UUID":"6b9484f9-983f-4b5c-bbbf-5d3ec0128bd9","Handler":null,"Name":"","Endpoint":""} Nov 01 22:23:54.822790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.822773 4981 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 22:23:54.823362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.823348 4981 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 22:23:54.824297 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:54.824279 4981 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 22:23:54.826968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.826916 4981 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal"] Nov 01 22:23:54.828152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.828138 4981 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.828435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.828117 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"bda61bce4694bad4d0a8e6759feac37d","Type":"ContainerStarted","Data":"4efb31777281f0fd89553d0cd588edb64286cdce0d10382dc2e94cd2f5167423"} Nov 01 22:23:54.828526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.828462 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"bda61bce4694bad4d0a8e6759feac37d","Type":"ContainerDied","Data":"bc19ee39eaf2d65b3712dbd0f6e8285ed9cb9a2a2ec2297b6889f61075b30d67"} Nov 01 22:23:54.828526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.828483 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"bda61bce4694bad4d0a8e6759feac37d","Type":"ContainerStarted","Data":"4746344fb054fd33dd7b4f5436eea22d07b6e5f7303d9d28dc6bfc2e014c95b3"} Nov 01 22:23:54.828526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.828203 4981 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.828698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.828544 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"9bc77fd8b25c4cfddf6fa5b50fb587e4","Type":"ContainerStarted","Data":"e543a73b6a39072e910b53060d36acb8d4f70cc851a2a004fb97032535af903d"} Nov 01 22:23:54.828698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.828566 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" event={"ID":"9bc77fd8b25c4cfddf6fa5b50fb587e4","Type":"ContainerStarted","Data":"0f3b84df1fe74af3b4f384ad27eb52a650860583c513a451465cf5f9895cfab7"} Nov 01 22:23:54.849805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.849775 4981 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:54.849910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.849809 4981 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:54.849910 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:54.849868 4981 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.850014 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:54.849966 4981 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.894621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.894593 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bc77fd8b25c4cfddf6fa5b50fb587e4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"9bc77fd8b25c4cfddf6fa5b50fb587e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.894751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.894633 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.894751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.894660 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.921824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.921801 4981 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 22:23:54.923949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.923929 4981 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 22:23:54.924057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.923963 4981 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 22:23:54.924057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.923977 4981 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-87.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 22:23:54.924057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.924028 4981 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.937595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.937549 4981 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.937700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.937624 4981 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.995391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.995333 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.995391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.995385 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.995564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.995422 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.995564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.995448 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bc77fd8b25c4cfddf6fa5b50fb587e4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"9bc77fd8b25c4cfddf6fa5b50fb587e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.995564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.995467 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda61bce4694bad4d0a8e6759feac37d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"bda61bce4694bad4d0a8e6759feac37d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:54.995564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:54.995486 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bc77fd8b25c4cfddf6fa5b50fb587e4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" (UID: \"9bc77fd8b25c4cfddf6fa5b50fb587e4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:55.660350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.660320 4981 apiserver.go:52] "Watching apiserver" Nov 01 22:23:55.693218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.693185 4981 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 22:23:55.693889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.693859 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-xglkp","openshift-cluster-node-tuning-operator/tuned-wrllx","openshift-dns/dns-default-5v8l4","openshift-multus/multus-gxf42","openshift-network-diagnostics/network-check-target-mpjqm","openshift-image-registry/node-ca-tnr7k","openshift-insights/insights-runtime-extractor-m9kr7","kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46","openshift-multus/multus-additional-cni-plugins-bbtw6","openshift-multus/network-metrics-daemon-x44vh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-bwft6","kube-system/global-pull-secret-syncer-wdkm9","openshift-console/downloads-6dd789c8b7-n87cr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk","openshift-image-registry/image-registry-7d7bc7f98b-p87lt","openshift-ingress-canary/ingress-canary-82z6g","openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp","openshift-console/console-85d9849f88-4ntzt","kube-system/konnectivity-agent-2rvph","openshift-dns/node-resolver-n8wtc","openshift-monitoring/node-exporter-fhtnf"] Nov 01 22:23:55.707214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.707194 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.709314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.709290 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 22:23:55.709893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.709865 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.711168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.711124 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9gzz5\"" Nov 01 22:23:55.711168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.711141 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nz8z8\"" Nov 01 22:23:55.711604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.711501 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-th659\"" Nov 01 22:23:55.711720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.711681 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z74s5\"" Nov 01 22:23:55.711720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.711687 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sq9j\"" Nov 01 22:23:55.711720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.711705 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 22:23:55.711846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.711738 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vwc2\"" Nov 01 22:23:55.712434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712412 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qzblw\"" Nov 01 22:23:55.712434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712425 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sw8fs\"" Nov 01 22:23:55.712873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712724 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 22:23:55.712873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712727 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdns4\"" Nov 01 22:23:55.712873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712766 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jvq4g\"" Nov 01 22:23:55.712873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712788 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sml6h\"" Nov 01 22:23:55.712873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712807 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 22:23:55.712873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.712851 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 22:23:55.713279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.713262 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 01 22:23:55.713388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.713282 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.713486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.713411 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j4zfh\"" Nov 01 22:23:55.714814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.714798 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bpr7j\"" Nov 01 22:23:55.752940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.752916 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 22:23:55.753068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.752962 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 01 22:23:55.754062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.754038 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.754538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.754518 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hhd78\"" Nov 01 22:23:55.754666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.754597 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 22:23:55.754963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.754942 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vfgt9\"" Nov 01 22:23:55.755097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755081 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-d7qxj\"" Nov 01 22:23:55.755198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755185 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 22:23:55.755246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755234 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 22:23:55.755279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755266 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5n9np\"" Nov 01 22:23:55.755307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755272 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 01 22:23:55.755754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755735 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 01 22:23:55.755854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755798 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 22:23:55.755854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755846 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 01 22:23:55.755994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.755974 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b85vc\"" Nov 01 22:23:55.756226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756205 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6kgdt\"" Nov 01 22:23:55.756295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756244 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fdq7t\"" Nov 01 22:23:55.756351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756245 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 01 22:23:55.756394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756385 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mvk6z\"" Nov 01 22:23:55.756459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756443 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 01 22:23:55.756526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756461 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 22:23:55.756568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756552 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 22:23:55.756781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756764 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 01 22:23:55.756880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.756826 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7bq7\"" Nov 01 22:23:55.775347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.775327 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 22:23:55.775824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.775808 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.775899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.775861 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.775899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.775870 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.776291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.776272 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.776525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.776512 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 22:23:55.777317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777213 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.777317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777266 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 22:23:55.777317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777279 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 22:23:55.777317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777280 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.777647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777628 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 22:23:55.777768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777747 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.777870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777791 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 22:23:55.777870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777751 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 01 22:23:55.777870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777792 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.777870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777813 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 22:23:55.777870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777871 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.778045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777896 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 22:23:55.778045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.777948 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.778646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778629 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.778646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778631 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.778764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778690 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 22:23:55.778764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778691 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 22:23:55.778764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778708 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.778764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778717 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 22:23:55.778764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778738 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 22:23:55.778764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778759 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.779000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.778959 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.779042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779001 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 22:23:55.779154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779137 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 22:23:55.779216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779178 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.779216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779208 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 22:23:55.779299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779211 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.779299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779228 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.779299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779259 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.779299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779265 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.779299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.779299 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.780166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.780148 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 22:23:55.780166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.780161 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 22:23:55.780345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.780172 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.780345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.780161 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 22:23:55.780345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.780250 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 22:23:55.785382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.785365 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 01 22:23:55.790008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.789992 4981 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 22:23:55.802298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802270 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.802410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802316 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.802410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802348 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d574c8-9d38-43ba-99ea-c24167bf452b-metrics-client-ca\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.802410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802376 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.802410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802403 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7jg\" (UniqueName: \"kubernetes.io/projected/31cc6987-5648-4628-a50a-6cf2f1606627-kube-api-access-4j7jg\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.802528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802429 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:55.802528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802460 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.802528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802482 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-netd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.802528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802511 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-socket-dir-parent\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.802692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802536 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/874207d9-9be6-4544-bc12-2bf935d7e477-config-volume\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.802692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802559 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-var-lib-kubelet\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.802692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802602 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-certificates\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.802692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802631 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-trusted-ca\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.802692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802657 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km45f\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-kube-api-access-km45f\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.802692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802680 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2w7\" (UniqueName: \"kubernetes.io/projected/874207d9-9be6-4544-bc12-2bf935d7e477-kube-api-access-2w2w7\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802702 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83ec6a1b-db92-4103-adec-8539c8bb86c2-ca-trust-extracted\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802725 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-installation-pull-secrets\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802749 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-var-lib-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802770 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-slash\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802792 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802821 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6gnk\" (UniqueName: \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802844 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbtt\" (UniqueName: \"kubernetes.io/projected/47eb4322-3268-4e50-80e0-35f2dd802b0b-kube-api-access-qrbtt\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802873 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-kubelet-config\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.802911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802907 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovn-node-metrics-cert\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802931 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46gn\" (UniqueName: \"kubernetes.io/projected/4e15bc3a-dc26-4b68-bba9-b670d741c952-kube-api-access-z46gn\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802955 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.802976 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-k8s-cni-cncf-io\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803003 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f130f8c5-0161-4ef2-8634-cc150d09ddf0-serviceca\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803029 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-multus-certs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803055 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887gs\" (UniqueName: \"kubernetes.io/projected/aa89357c-1284-4f7a-b319-fcdb18edb560-kube-api-access-887gs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803082 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803113 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-script-lib\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803160 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-registration-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803186 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.803221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803215 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803246 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbpk\" (UniqueName: \"kubernetes.io/projected/e91e58ba-dbc6-49da-b601-22edc7b6525e-kube-api-access-fdbpk\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803270 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-dbus\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803300 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-bound-sa-token\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803327 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e15bc3a-dc26-4b68-bba9-b670d741c952-cert\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803353 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-cnibin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803379 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-etc-selinux\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803407 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803435 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-textfile\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803456 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysconfig\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803479 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cnibin\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803503 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-data-volume\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.803531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803524 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-node-log\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803547 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7eeac783-1128-4402-8404-5962939c1b63-iptables-alerter-script\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803588 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-bin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803613 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c512618-f7bd-4210-90f9-e525e4a8a1b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2k4xp\" (UID: \"5c512618-f7bd-4210-90f9-e525e4a8a1b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803637 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803662 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803702 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7025fde7-1830-4d2e-9780-a4c0a8195d0f-konnectivity-ca\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803729 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-env-overrides\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803753 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803776 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803801 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82s88\" (UniqueName: \"kubernetes.io/projected/7eeac783-1128-4402-8404-5962939c1b63-kube-api-access-82s88\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803827 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pchns\" (UniqueName: \"kubernetes.io/projected/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-api-access-pchns\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803857 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-root\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803882 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803907 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7025fde7-1830-4d2e-9780-a4c0a8195d0f-agent-certs\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:23:55.803947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803931 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-systemd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803958 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-device-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.803983 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804007 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-netns\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804031 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eeac783-1128-4402-8404-5962939c1b63-host-slash\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804057 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804088 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8f9\" (UniqueName: \"kubernetes.io/projected/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-api-access-dm8f9\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804114 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-config\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804141 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s28ks\" (UniqueName: \"kubernetes.io/projected/8deca68d-1a0b-4841-9c91-0682262d59c7-kube-api-access-s28ks\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804165 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-system-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804189 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-sys-fs\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804215 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5z4\" (UniqueName: \"kubernetes.io/projected/a4ce9767-19f2-42a8-afe1-ad361726b8b9-kube-api-access-xk5z4\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804240 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55hgz\" (UniqueName: \"kubernetes.io/projected/948069e6-a5b8-407d-a12b-198a76a2e783-kube-api-access-55hgz\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804263 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-host\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804287 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zkk\" (UniqueName: \"kubernetes.io/projected/548c3a65-3217-4609-b266-b7eacca269a3-kube-api-access-56zkk\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804315 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpmt\" (UniqueName: \"kubernetes.io/projected/0f3425f3-64ee-4134-b096-10e48c0c5faa-kube-api-access-lcpmt\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804344 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804371 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/948069e6-a5b8-407d-a12b-198a76a2e783-hosts-file\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804397 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948069e6-a5b8-407d-a12b-198a76a2e783-tmp-dir\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804421 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-conf\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804445 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-systemd\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804469 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-tmp\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804498 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-systemd-units\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804524 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-log-socket\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804555 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-conf-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804596 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-run\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.804635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804624 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31cc6987-5648-4628-a50a-6cf2f1606627-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804666 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804691 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-wtmp\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804717 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-kubernetes\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804742 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-etc-tuned\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804766 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-tls\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804794 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cc6987-5648-4628-a50a-6cf2f1606627-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804820 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-daemon-config\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804844 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-sys\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804878 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-etc-kubernetes\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804902 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f130f8c5-0161-4ef2-8634-cc150d09ddf0-host\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804925 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5b8\" (UniqueName: \"kubernetes.io/projected/f130f8c5-0161-4ef2-8634-cc150d09ddf0-kube-api-access-cl5b8\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804954 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.804980 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805008 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-lib-modules\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805035 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-netns\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805060 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805085 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-system-cni-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805111 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-tls\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805136 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-sys\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805159 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-ovn\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805190 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805214 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-bin\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805239 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-cni-binary-copy\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805267 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-hostroot\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805294 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/874207d9-9be6-4544-bc12-2bf935d7e477-tmp-dir\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805322 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-kubelet\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805352 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-kubelet\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805381 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqk7\" (UniqueName: \"kubernetes.io/projected/e0d574c8-9d38-43ba-99ea-c24167bf452b-kube-api-access-mfqk7\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805409 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805462 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805511 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-crio-socket\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805541 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31cc6987-5648-4628-a50a-6cf2f1606627-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805585 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deca68d-1a0b-4841-9c91-0682262d59c7-tmp\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805770 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-etc-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805804 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-multus\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805825 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805843 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-socket-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805862 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-modprobe-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805887 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/874207d9-9be6-4544-bc12-2bf935d7e477-metrics-tls\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.805906 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-image-registry-private-configuration\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.806092 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-image-registry-private-configuration\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.806162 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.806395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.806300 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.806478 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb4322-3268-4e50-80e0-35f2dd802b0b-metrics-certs\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.806645 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/874207d9-9be6-4544-bc12-2bf935d7e477-config-volume\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.806835 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807059 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-data-volume\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807147 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807235 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7eeac783-1128-4402-8404-5962939c1b63-iptables-alerter-script\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807274 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7025fde7-1830-4d2e-9780-a4c0a8195d0f-konnectivity-ca\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807338 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-script-lib\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807344 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7025fde7-1830-4d2e-9780-a4c0a8195d0f-agent-certs\") pod \"konnectivity-agent-2rvph\" (UID: \"7025fde7-1830-4d2e-9780-a4c0a8195d0f\") " pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807427 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-env-overrides\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807607 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5c512618-f7bd-4210-90f9-e525e4a8a1b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2k4xp\" (UID: \"5c512618-f7bd-4210-90f9-e525e4a8a1b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807640 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807695 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7caa0db8-4f8f-4848-a2a6-845f20c9a888-original-pull-secret\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807731 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807818 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.807847 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808074 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovnkube-config\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.808292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808142 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-certificates\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808385 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83ec6a1b-db92-4103-adec-8539c8bb86c2-trusted-ca\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808394 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83ec6a1b-db92-4103-adec-8539c8bb86c2-installation-pull-secrets\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808445 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808471 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deca68d-1a0b-4841-9c91-0682262d59c7-tmp\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808516 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-cni-binary-copy\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808669 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808738 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/874207d9-9be6-4544-bc12-2bf935d7e477-tmp-dir\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808814 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-registry-tls\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808869 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-tmp\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808928 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/548c3a65-3217-4609-b266-b7eacca269a3-etc-tuned\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808949 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e91e58ba-dbc6-49da-b601-22edc7b6525e-ovn-node-metrics-cert\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.808984 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83ec6a1b-db92-4103-adec-8539c8bb86c2-ca-trust-extracted\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.809118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.809017 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948069e6-a5b8-407d-a12b-198a76a2e783-tmp-dir\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:23:55.809540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.809209 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-daemon-config\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.809540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.809243 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.809540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.809513 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f130f8c5-0161-4ef2-8634-cc150d09ddf0-serviceca\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:23:55.809713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.809693 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/874207d9-9be6-4544-bc12-2bf935d7e477-metrics-tls\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.809891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.809879 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e15bc3a-dc26-4b68-bba9-b670d741c952-cert\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:55.812688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.812670 4981 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:55.812805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.812759 4981 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:55.818928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.818907 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-bound-sa-token\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.819014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.818945 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zkk\" (UniqueName: \"kubernetes.io/projected/548c3a65-3217-4609-b266-b7eacca269a3-kube-api-access-56zkk\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.819088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.819068 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82s88\" (UniqueName: \"kubernetes.io/projected/7eeac783-1128-4402-8404-5962939c1b63-kube-api-access-82s88\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:23:55.819259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.819240 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbtt\" (UniqueName: \"kubernetes.io/projected/47eb4322-3268-4e50-80e0-35f2dd802b0b-kube-api-access-qrbtt\") pod \"network-metrics-daemon-x44vh\" (UID: \"47eb4322-3268-4e50-80e0-35f2dd802b0b\") " pod="openshift-multus/network-metrics-daemon-x44vh" Nov 01 22:23:55.819785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.819766 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km45f\" (UniqueName: \"kubernetes.io/projected/83ec6a1b-db92-4103-adec-8539c8bb86c2-kube-api-access-km45f\") pod \"image-registry-7d7bc7f98b-p87lt\" (UID: \"83ec6a1b-db92-4103-adec-8539c8bb86c2\") " pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:55.820530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.820511 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s28ks\" (UniqueName: \"kubernetes.io/projected/8deca68d-1a0b-4841-9c91-0682262d59c7-kube-api-access-s28ks\") pod \"downloads-6dd789c8b7-n87cr\" (UID: \"8deca68d-1a0b-4841-9c91-0682262d59c7\") " pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:23:55.821224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.821203 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8f9\" (UniqueName: \"kubernetes.io/projected/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-kube-api-access-dm8f9\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.821625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.821604 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2w7\" (UniqueName: \"kubernetes.io/projected/874207d9-9be6-4544-bc12-2bf935d7e477-kube-api-access-2w2w7\") pod \"dns-default-5v8l4\" (UID: \"874207d9-9be6-4544-bc12-2bf935d7e477\") " pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:55.821732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.821666 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46gn\" (UniqueName: \"kubernetes.io/projected/4e15bc3a-dc26-4b68-bba9-b670d741c952-kube-api-access-z46gn\") pod \"ingress-canary-82z6g\" (UID: \"4e15bc3a-dc26-4b68-bba9-b670d741c952\") " pod="openshift-ingress-canary/ingress-canary-82z6g" Nov 01 22:23:55.822233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.822215 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpmt\" (UniqueName: \"kubernetes.io/projected/0f3425f3-64ee-4134-b096-10e48c0c5faa-kube-api-access-lcpmt\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.822319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.822269 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5z4\" (UniqueName: \"kubernetes.io/projected/a4ce9767-19f2-42a8-afe1-ad361726b8b9-kube-api-access-xk5z4\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.822863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.822843 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6gnk\" (UniqueName: \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\") pod \"console-85d9849f88-4ntzt\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:55.822931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.822867 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbpk\" (UniqueName: \"kubernetes.io/projected/e91e58ba-dbc6-49da-b601-22edc7b6525e-kube-api-access-fdbpk\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.823122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.823100 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887gs\" (UniqueName: \"kubernetes.io/projected/aa89357c-1284-4f7a-b319-fcdb18edb560-kube-api-access-887gs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.823181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.823162 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55hgz\" (UniqueName: \"kubernetes.io/projected/948069e6-a5b8-407d-a12b-198a76a2e783-kube-api-access-55hgz\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:23:55.823535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.823515 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5b8\" (UniqueName: \"kubernetes.io/projected/f130f8c5-0161-4ef2-8634-cc150d09ddf0-kube-api-access-cl5b8\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:23:55.823900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.823872 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz2q\" (UniqueName: \"kubernetes.io/projected/3d4713d5-cc55-4d7e-86f8-f1e17f3b3858-kube-api-access-wwz2q\") pod \"network-check-target-mpjqm\" (UID: \"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858\") " pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:55.827638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.827617 4981 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:55.827716 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:55.827682 4981 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:55.831048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.831031 4981 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 22:23:55.831154 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:55.831070 4981 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-87.us-east-2.compute.internal" Nov 01 22:23:55.869534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.869508 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.906586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906543 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-node-log\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.906719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906593 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-bin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.906719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906619 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.906719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906640 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.906719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906663 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-node-log\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.906719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906681 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-bin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.906719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906665 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchns\" (UniqueName: \"kubernetes.io/projected/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-api-access-pchns\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906727 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-root\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906753 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906779 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-systemd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906803 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-device-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906804 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-root\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906827 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906854 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-netns\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906879 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eeac783-1128-4402-8404-5962939c1b63-host-slash\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906883 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-device-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906909 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-system-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906857 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-systemd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906958 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-sys-fs\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906990 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7eeac783-1128-4402-8404-5962939c1b63-host-slash\") pod \"iptables-alerter-bwft6\" (UID: \"7eeac783-1128-4402-8404-5962939c1b63\") " pod="openshift-network-operator/iptables-alerter-bwft6" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.906957 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-system-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.907001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907003 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-host\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907029 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-sys-fs\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907039 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907070 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/948069e6-a5b8-407d-a12b-198a76a2e783-hosts-file\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907079 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-host\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907085 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907113 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-netns\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907124 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/948069e6-a5b8-407d-a12b-198a76a2e783-hosts-file\") pod \"node-resolver-n8wtc\" (UID: \"948069e6-a5b8-407d-a12b-198a76a2e783\") " pod="openshift-dns/node-resolver-n8wtc" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907115 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-conf\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907117 4981 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907161 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-systemd\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907187 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-systemd-units\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907213 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-log-socket\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907236 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-conf-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907258 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-run\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907264 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-systemd-units\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907271 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-conf\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907267 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-systemd\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907300 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-log-socket\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907305 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-conf-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907286 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31cc6987-5648-4628-a50a-6cf2f1606627-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907369 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-wtmp\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907415 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-kubernetes\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907418 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-run\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907455 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cc6987-5648-4628-a50a-6cf2f1606627-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907481 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-kubernetes\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907504 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-sys\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907528 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-etc-kubernetes\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907559 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-sys\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907569 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f130f8c5-0161-4ef2-8634-cc150d09ddf0-host\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907618 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-wtmp\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907635 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-lib-modules\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907669 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f130f8c5-0161-4ef2-8634-cc150d09ddf0-host\") pod \"node-ca-tnr7k\" (UID: \"f130f8c5-0161-4ef2-8634-cc150d09ddf0\") " pod="openshift-image-registry/node-ca-tnr7k" Nov 01 22:23:55.907695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907624 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-etc-kubernetes\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907785 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-lib-modules\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907803 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-netns\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907861 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-netns\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907899 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.907976 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908051 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-system-cni-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908157 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-system-cni-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908171 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908194 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-tls\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908597 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-sys\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908369 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31cc6987-5648-4628-a50a-6cf2f1606627-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908684 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-sys\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908724 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-ovn\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908746 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908801 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-ovn\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908839 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-bin\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908865 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-hostroot\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908983 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-hostroot\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908945 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-bin\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908987 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-kubelet\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.908909 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-run-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909039 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-kubelet\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909065 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqk7\" (UniqueName: \"kubernetes.io/projected/e0d574c8-9d38-43ba-99ea-c24167bf452b-kube-api-access-mfqk7\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909074 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-kubelet\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909089 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909113 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-crio-socket\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31cc6987-5648-4628-a50a-6cf2f1606627-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.909160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909163 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-etc-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909184 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-multus\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909209 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909233 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-socket-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909256 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-modprobe-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909282 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d574c8-9d38-43ba-99ea-c24167bf452b-metrics-client-ca\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909293 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-etc-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909364 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909391 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7jg\" (UniqueName: \"kubernetes.io/projected/31cc6987-5648-4628-a50a-6cf2f1606627-kube-api-access-4j7jg\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909415 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909438 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-netd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909463 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-socket-dir-parent\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909489 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-var-lib-kubelet\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909521 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-var-lib-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909551 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-slash\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909596 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-kubelet-config\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909625 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909649 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-k8s-cni-cncf-io\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909673 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-multus-certs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909701 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909748 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-registration-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909776 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-dbus\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909814 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-cnibin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909852 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-etc-selinux\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909860 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909877 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909864 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909897 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d9cf0fb4-ef5d-4921-9c01-8830c386dce8-crio-socket\") pod \"insights-runtime-extractor-m9kr7\" (UID: \"d9cf0fb4-ef5d-4921-9c01-8830c386dce8\") " pod="openshift-insights/insights-runtime-extractor-m9kr7" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909904 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-textfile\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909934 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysconfig\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909936 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.909958 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cnibin\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910000 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-slash\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910047 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-cnibin\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910089 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-var-lib-cni-multus\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910153 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-dbus\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910157 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7caa0db8-4f8f-4848-a2a6-845f20c9a888-kubelet-config\") pod \"global-pull-secret-syncer-wdkm9\" (UID: \"7caa0db8-4f8f-4848-a2a6-845f20c9a888\") " pod="kube-system/global-pull-secret-syncer-wdkm9" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910205 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-cnibin\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910208 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-registration-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910256 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-etc-selinux\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910266 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-cni-dir\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910294 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910305 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-k8s-cni-cncf-io\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910308 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a4ce9767-19f2-42a8-afe1-ad361726b8b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bbtw6\" (UID: \"a4ce9767-19f2-42a8-afe1-ad361726b8b9\") " pod="openshift-multus/multus-additional-cni-plugins-bbtw6" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910355 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910367 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysconfig\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.910416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910394 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-host-run-multus-certs\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910453 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa89357c-1284-4f7a-b319-fcdb18edb560-multus-socket-dir-parent\") pod \"multus-gxf42\" (UID: \"aa89357c-1284-4f7a-b319-fcdb18edb560\") " pod="openshift-multus/multus-gxf42" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910495 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-run-ovn-kubernetes\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910532 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-cni-netd\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910559 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-textfile\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910602 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-var-lib-kubelet\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910620 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910646 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-var-lib-openvswitch\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910657 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-modprobe-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910685 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f3425f3-64ee-4134-b096-10e48c0c5faa-socket-dir\") pod \"aws-ebs-csi-driver-node-gglwk\" (UID: \"0f3425f3-64ee-4134-b096-10e48c0c5faa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gglwk" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.910710 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/548c3a65-3217-4609-b266-b7eacca269a3-etc-sysctl-d\") pod \"tuned-wrllx\" (UID: \"548c3a65-3217-4609-b266-b7eacca269a3\") " pod="openshift-cluster-node-tuning-operator/tuned-wrllx" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.911015 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d574c8-9d38-43ba-99ea-c24167bf452b-metrics-client-ca\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.911343 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0d574c8-9d38-43ba-99ea-c24167bf452b-node-exporter-tls\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.912310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.911384 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e91e58ba-dbc6-49da-b601-22edc7b6525e-host-kubelet\") pod \"ovnkube-node-xglkp\" (UID: \"e91e58ba-dbc6-49da-b601-22edc7b6525e\") " pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:23:55.912713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.912325 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31cc6987-5648-4628-a50a-6cf2f1606627-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.912713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.912372 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31cc6987-5648-4628-a50a-6cf2f1606627-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.917650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.917626 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqk7\" (UniqueName: \"kubernetes.io/projected/e0d574c8-9d38-43ba-99ea-c24167bf452b-kube-api-access-mfqk7\") pod \"node-exporter-fhtnf\" (UID: \"e0d574c8-9d38-43ba-99ea-c24167bf452b\") " pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:55.917872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.917856 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchns\" (UniqueName: \"kubernetes.io/projected/2e4ab4e4-564d-4403-bc0e-802ba2a2d29f-kube-api-access-pchns\") pod \"kube-state-metrics-5dc8d7d89b-bvp46\" (UID: \"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:55.918419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.918389 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7jg\" (UniqueName: \"kubernetes.io/projected/31cc6987-5648-4628-a50a-6cf2f1606627-kube-api-access-4j7jg\") pod \"openshift-state-metrics-7bd7489957-m2b87\" (UID: \"31cc6987-5648-4628-a50a-6cf2f1606627\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:55.951632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.951606 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:55.997403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:55.997377 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:23:56.026482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.026452 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" Nov 01 22:23:56.033676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.033650 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.033783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.033758 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" Nov 01 22:23:56.040623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.040599 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhtnf" Nov 01 22:23:56.054933 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:56.054898 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d574c8_9d38_43ba_99ea_c24167bf452b.slice/crio-b40abd2f14d26ef1fe265852c2a88465b558e77833f98ce9834fa5446ea78077 WatchSource:0}: Error finding container b40abd2f14d26ef1fe265852c2a88465b558e77833f98ce9834fa5446ea78077: Status 404 returned error can't find the container with id b40abd2f14d26ef1fe265852c2a88465b558e77833f98ce9834fa5446ea78077 Nov 01 22:23:56.148346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.148319 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d7bc7f98b-p87lt" Nov 01 22:23:56.152717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.152689 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.299484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.299437 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.310877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.310851 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87"] Nov 01 22:23:56.314082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.314055 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46"] Nov 01 22:23:56.315010 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:56.314979 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cc6987_5648_4628_a50a_6cf2f1606627.slice/crio-ea809a4990ef38be407441b4f613b653c19f33fcb32635e28133ddcd09b26311 WatchSource:0}: Error finding container ea809a4990ef38be407441b4f613b653c19f33fcb32635e28133ddcd09b26311: Status 404 returned error can't find the container with id ea809a4990ef38be407441b4f613b653c19f33fcb32635e28133ddcd09b26311 Nov 01 22:23:56.317869 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:56.317839 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4ab4e4_564d_4403_bc0e_802ba2a2d29f.slice/crio-92c29a2bda98f00c631718b8bab937081c0b81811fefcc9d37f23f1353d403a1 WatchSource:0}: Error finding container 92c29a2bda98f00c631718b8bab937081c0b81811fefcc9d37f23f1353d403a1: Status 404 returned error can't find the container with id 92c29a2bda98f00c631718b8bab937081c0b81811fefcc9d37f23f1353d403a1 Nov 01 22:23:56.400560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.400530 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.501478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.501424 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.640508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.640482 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.770475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.770437 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:56.818211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.818173 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" event={"ID":"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f","Type":"ContainerStarted","Data":"92c29a2bda98f00c631718b8bab937081c0b81811fefcc9d37f23f1353d403a1"} Nov 01 22:23:56.820317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.820278 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d9849f88-4ntzt" event={"ID":"2fe7ab41-3bca-400b-9a55-ed06fb29c216","Type":"ContainerStarted","Data":"45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0"} Nov 01 22:23:56.822334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.822310 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhtnf" event={"ID":"e0d574c8-9d38-43ba-99ea-c24167bf452b","Type":"ContainerStarted","Data":"b40abd2f14d26ef1fe265852c2a88465b558e77833f98ce9834fa5446ea78077"} Nov 01 22:23:56.824914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.824888 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" event={"ID":"31cc6987-5648-4628-a50a-6cf2f1606627","Type":"ContainerStarted","Data":"82c8554098cf6e6106199587c98fcf55003ee0ab552246d0e2c2995d1c451406"} Nov 01 22:23:56.825035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.824942 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" event={"ID":"31cc6987-5648-4628-a50a-6cf2f1606627","Type":"ContainerStarted","Data":"eb27571b87516929bdeff7f45a87dc9b88198a5b5d0959a95824c65a48ed5099"} Nov 01 22:23:56.825035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.824973 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" event={"ID":"31cc6987-5648-4628-a50a-6cf2f1606627","Type":"ContainerStarted","Data":"ea809a4990ef38be407441b4f613b653c19f33fcb32635e28133ddcd09b26311"} Nov 01 22:23:56.826897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.826823 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mpjqm" event={"ID":"3d4713d5-cc55-4d7e-86f8-f1e17f3b3858","Type":"ContainerStarted","Data":"fd80964772780039f4ef329b9c36408945d0efab4f04ebbd2f2f017283416b2d"} Nov 01 22:23:56.970313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:56.970277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:57.623787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.623745 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6887556c4f-ppfhf"] Nov 01 22:23:57.630854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.630828 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.637757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.637725 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 22:23:57.640089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.640067 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6887556c4f-ppfhf"] Nov 01 22:23:57.640772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.640739 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 22:23:57.640920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.640828 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 22:23:57.640988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.640966 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d3a43dv09ba1t\"" Nov 01 22:23:57.726593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7900c2-b948-46db-a1fa-535cf450e73b-tmp\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.726771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726606 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.726771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726645 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-tls\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.726771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726671 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.726771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726723 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e7900c2-b948-46db-a1fa-535cf450e73b-metrics-client-ca\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.726771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726746 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-grpc-tls\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.727036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726782 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.727036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726811 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.727036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.726841 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqxw\" (UniqueName: \"kubernetes.io/projected/3e7900c2-b948-46db-a1fa-535cf450e73b-kube-api-access-6fqxw\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.737895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.737866 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 22:23:57.738041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.737935 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 22:23:57.738041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.737945 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v8hfd\"" Nov 01 22:23:57.827327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827288 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqxw\" (UniqueName: \"kubernetes.io/projected/3e7900c2-b948-46db-a1fa-535cf450e73b-kube-api-access-6fqxw\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827368 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7900c2-b948-46db-a1fa-535cf450e73b-tmp\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827401 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827433 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-tls\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827458 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827513 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e7900c2-b948-46db-a1fa-535cf450e73b-metrics-client-ca\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827538 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-grpc-tls\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827595 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.827802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.827623 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.829101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.828743 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e7900c2-b948-46db-a1fa-535cf450e73b-metrics-client-ca\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.829101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.829012 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7900c2-b948-46db-a1fa-535cf450e73b-tmp\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.837479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.836841 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-grpc-tls\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.837479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.837388 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqxw\" (UniqueName: \"kubernetes.io/projected/3e7900c2-b948-46db-a1fa-535cf450e73b-kube-api-access-6fqxw\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.837726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.837478 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-tls\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.839538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.839490 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.840054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.840014 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.841166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.841107 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.841669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.841626 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e7900c2-b948-46db-a1fa-535cf450e73b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6887556c4f-ppfhf\" (UID: \"3e7900c2-b948-46db-a1fa-535cf450e73b\") " pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.959499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.959465 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:23:57.964643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.964618 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:23:57.972947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.971791 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dccbd4987-22x68"] Nov 01 22:23:57.975964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.975418 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:57.990125 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.990100 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dccbd4987-22x68"] Nov 01 22:23:57.995077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:57.995035 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.029834 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-trusted-ca-bundle\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.029881 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h486v\" (UniqueName: \"kubernetes.io/projected/c2beea1c-4d22-435c-8214-283eca213a15-kube-api-access-h486v\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.029923 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-service-ca\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.029939 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-oauth-serving-cert\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.029983 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-serving-cert\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.030010 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-console-config\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.030026 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2beea1c-4d22-435c-8214-283eca213a15-tmp\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.031069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.030055 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-oauth-config\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131043 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2beea1c-4d22-435c-8214-283eca213a15-tmp\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.130569 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2beea1c-4d22-435c-8214-283eca213a15-tmp\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131396 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-oauth-config\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131481 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-trusted-ca-bundle\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131508 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h486v\" (UniqueName: \"kubernetes.io/projected/c2beea1c-4d22-435c-8214-283eca213a15-kube-api-access-h486v\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131539 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-service-ca\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131566 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-oauth-serving-cert\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131629 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-serving-cert\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.131720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.131668 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-console-config\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.132313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.132285 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-console-config\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.132881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.132857 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-service-ca\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.133456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.133432 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-oauth-serving-cert\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.133912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.133892 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-trusted-ca-bundle\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.134777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.134757 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-oauth-config\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.136274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.136233 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-serving-cert\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.141106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.140727 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h486v\" (UniqueName: \"kubernetes.io/projected/c2beea1c-4d22-435c-8214-283eca213a15-kube-api-access-h486v\") pod \"console-7dccbd4987-22x68\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.150175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.150150 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:58.236050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.235256 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5v8l4" Nov 01 22:23:58.250988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.250895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:58.277905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.277875 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6887556c4f-ppfhf"] Nov 01 22:23:58.281165 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:58.281126 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7900c2_b948_46db_a1fa_535cf450e73b.slice/crio-27046cb158d99a911d2935f3a5149ce0a2048ddaaae2c27f48718e0da66b1b48 WatchSource:0}: Error finding container 27046cb158d99a911d2935f3a5149ce0a2048ddaaae2c27f48718e0da66b1b48: Status 404 returned error can't find the container with id 27046cb158d99a911d2935f3a5149ce0a2048ddaaae2c27f48718e0da66b1b48 Nov 01 22:23:58.300784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.300670 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:23:58.619183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.619108 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dccbd4987-22x68"] Nov 01 22:23:58.622931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.622909 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:58.706949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.706918 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:58.783273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.783224 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:23:58.791340 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:23:58.791299 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2beea1c_4d22_435c_8214_283eca213a15.slice/crio-661197cba491feebe1d16441babbc6e79cc66ca6674221df58b4265e8d7476ca WatchSource:0}: Error finding container 661197cba491feebe1d16441babbc6e79cc66ca6674221df58b4265e8d7476ca: Status 404 returned error can't find the container with id 661197cba491feebe1d16441babbc6e79cc66ca6674221df58b4265e8d7476ca Nov 01 22:23:58.837989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.837953 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" event={"ID":"3e7900c2-b948-46db-a1fa-535cf450e73b","Type":"ContainerStarted","Data":"27046cb158d99a911d2935f3a5149ce0a2048ddaaae2c27f48718e0da66b1b48"} Nov 01 22:23:58.840409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.840380 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" event={"ID":"31cc6987-5648-4628-a50a-6cf2f1606627","Type":"ContainerStarted","Data":"4d00a73f4b4d32bfe13eb89569ab506c54ccae5933e54244f53446a631ec7863"} Nov 01 22:23:58.841645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.841625 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccbd4987-22x68" event={"ID":"c2beea1c-4d22-435c-8214-283eca213a15","Type":"ContainerStarted","Data":"661197cba491feebe1d16441babbc6e79cc66ca6674221df58b4265e8d7476ca"} Nov 01 22:23:58.860606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:58.860547 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m2b87" podStartSLOduration=2.225893956 podStartE2EDuration="3.860531966s" totalImagesPullingTime="1.63463801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:58.859155126 +0000 UTC m=+4.287033060" watchObservedRunningTime="2026-11-01 22:23:58.860531966 +0000 UTC m=+4.288409897" Nov 01 22:23:59.592413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.592340 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:23:59.847124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.847025 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" event={"ID":"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f","Type":"ContainerStarted","Data":"a3c5e30b4cbfaab6f8e6cb3538ff6587942e9a20e94206406249196f87d7e992"} Nov 01 22:23:59.847124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.847110 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" event={"ID":"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f","Type":"ContainerStarted","Data":"5dd9ae04091e17a8837f6596eeb88cae1038a83800cb73caa0c7d25c447160b6"} Nov 01 22:23:59.847656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.847138 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" event={"ID":"2e4ab4e4-564d-4403-bc0e-802ba2a2d29f","Type":"ContainerStarted","Data":"e973cd4597092c9da4ac377e859919503c197342ff626da1955e6436adb473cc"} Nov 01 22:23:59.848551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.848529 4981 generic.go:356] "Generic (PLEG): container finished" podID="e0d574c8-9d38-43ba-99ea-c24167bf452b" containerID="4ff42d5c19b4791412070c8a3b28096b2510b49be835023bd0a0309942c08e39" exitCode=0 Nov 01 22:23:59.848654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.848614 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhtnf" event={"ID":"e0d574c8-9d38-43ba-99ea-c24167bf452b","Type":"ContainerDied","Data":"4ff42d5c19b4791412070c8a3b28096b2510b49be835023bd0a0309942c08e39"} Nov 01 22:23:59.850194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.850172 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccbd4987-22x68" event={"ID":"c2beea1c-4d22-435c-8214-283eca213a15","Type":"ContainerStarted","Data":"fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c"} Nov 01 22:23:59.891968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.891876 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bvp46" podStartSLOduration=2.191826419 podStartE2EDuration="4.891861376s" totalImagesPullingTime="2.700034957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:59.868372258 +0000 UTC m=+5.296250190" watchObservedRunningTime="2026-11-01 22:23:59.891861376 +0000 UTC m=+5.319739305" Nov 01 22:23:59.892375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:23:59.892347 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7dccbd4987-22x68" podStartSLOduration=2.892340552 podStartE2EDuration="2.892340552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:59.89138445 +0000 UTC m=+5.319262380" watchObservedRunningTime="2026-11-01 22:23:59.892340552 +0000 UTC m=+5.320218481" Nov 01 22:23:59.958775 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:59.958737 4981 log.go:32] "PullImage from image service failed" err="rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36c3da5821b9e51334e9c523aca7fca6a9ef8b853fd4f238b908b086b129619d" Nov 01 22:23:59.958964 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:59.958943 4981 kuberuntime_manager.go:1707] "Unhandled Error" err="container download-server start failed in pod downloads-6dd789c8b7-n87cr_openshift-console(8deca68d-1a0b-4841-9c91-0682262d59c7): ErrImagePull: rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" logger="UnhandledError" Nov 01 22:23:59.959042 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:23:59.958982 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"download-server\" with ErrImagePull: \"rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled\"" pod="openshift-console/downloads-6dd789c8b7-n87cr" podUID="8deca68d-1a0b-4841-9c91-0682262d59c7" Nov 01 22:24:00.000510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.000493 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dccbd4987-22x68_c2beea1c-4d22-435c-8214-283eca213a15/console/0.log" Nov 01 22:24:00.201494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.201468 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85d9849f88-4ntzt_2fe7ab41-3bca-400b-9a55-ed06fb29c216/console/0.log" Nov 01 22:24:00.401437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.401408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:00.476505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.476431 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:00.600985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.600961 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5v8l4_874207d9-9be6-4544-bc12-2bf935d7e477/dns/0.log" Nov 01 22:24:00.802671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.802598 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5v8l4_874207d9-9be6-4544-bc12-2bf935d7e477/kube-rbac-proxy/0.log" Nov 01 22:24:00.856910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.856808 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhtnf" event={"ID":"e0d574c8-9d38-43ba-99ea-c24167bf452b","Type":"ContainerStarted","Data":"ded63ea7b7793b4ee6acbe9271fc546c554ad88bebf5ee1491ec95f51baed32d"} Nov 01 22:24:00.856910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.856873 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhtnf" event={"ID":"e0d574c8-9d38-43ba-99ea-c24167bf452b","Type":"ContainerStarted","Data":"73d1d60a4786fc77c54738521901354fd24aed583797b9c77b0f07682551a65b"} Nov 01 22:24:00.863571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.863545 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69d449f8cd-dgrw9"] Nov 01 22:24:00.866232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.866209 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.869748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.869728 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 22:24:00.869881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.869760 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-x4xml\"" Nov 01 22:24:00.869881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.869826 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 22:24:00.869979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.869885 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 22:24:00.872546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.872529 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 22:24:00.873979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.873955 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 22:24:00.879759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.879536 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 22:24:00.883727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.883710 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69d449f8cd-dgrw9"] Nov 01 22:24:00.902071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.902023 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fhtnf" podStartSLOduration=2.834860498 podStartE2EDuration="5.902006905s" totalImagesPullingTime="2.96019439s" totalInitContainerRuntime="106.952017ms" isStatefulPod=true podCreationTimestamp="2026-11-01 22:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:00.88284883 +0000 UTC m=+6.310726752" watchObservedRunningTime="2026-11-01 22:24:00.902006905 +0000 UTC m=+6.329884838" Nov 01 22:24:00.957777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.957299 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.957777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.957394 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-serving-certs-ca-bundle\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.957777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.957423 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnh5q\" (UniqueName: \"kubernetes.io/projected/c95c3ce6-d144-4423-94fc-65f47402f703-kube-api-access-fnh5q\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.957777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.957519 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-metrics-client-ca\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.957777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.957632 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-telemeter-client-tls\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.958243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.957935 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-secret-telemeter-client\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.958243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.958010 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-federate-client-tls\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:00.958243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:00.958050 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058518 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058565 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-serving-certs-ca-bundle\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058598 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnh5q\" (UniqueName: \"kubernetes.io/projected/c95c3ce6-d144-4423-94fc-65f47402f703-kube-api-access-fnh5q\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058722 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-metrics-client-ca\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058762 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-telemeter-client-tls\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058808 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-secret-telemeter-client\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058835 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-federate-client-tls\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.058902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.058851 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.059381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.059345 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-serving-certs-ca-bundle\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.059503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.059422 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-metrics-client-ca\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.059675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.059652 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c95c3ce6-d144-4423-94fc-65f47402f703-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.061407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.061388 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-federate-client-tls\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.061494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.061412 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.061747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.061729 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-telemeter-client-tls\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.061789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.061753 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c95c3ce6-d144-4423-94fc-65f47402f703-secret-telemeter-client\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.066453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.066432 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnh5q\" (UniqueName: \"kubernetes.io/projected/c95c3ce6-d144-4423-94fc-65f47402f703-kube-api-access-fnh5q\") pod \"telemeter-client-69d449f8cd-dgrw9\" (UID: \"c95c3ce6-d144-4423-94fc-65f47402f703\") " pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.185938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.185905 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" Nov 01 22:24:01.462320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.462229 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69d449f8cd-dgrw9"] Nov 01 22:24:01.466308 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:24:01.466281 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95c3ce6_d144_4423_94fc_65f47402f703.slice/crio-925f410a151f7e81c1c729c168c212dc19e49f90669ce646bd177c06e4eae976 WatchSource:0}: Error finding container 925f410a151f7e81c1c729c168c212dc19e49f90669ce646bd177c06e4eae976: Status 404 returned error can't find the container with id 925f410a151f7e81c1c729c168c212dc19e49f90669ce646bd177c06e4eae976 Nov 01 22:24:01.704939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.704911 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:24:01.802030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.801962 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n8wtc_948069e6-a5b8-407d-a12b-198a76a2e783/dns-node-resolver/0.log" Nov 01 22:24:01.861734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.861696 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" event={"ID":"c95c3ce6-d144-4423-94fc-65f47402f703","Type":"ContainerStarted","Data":"925f410a151f7e81c1c729c168c212dc19e49f90669ce646bd177c06e4eae976"} Nov 01 22:24:01.927864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.927825 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:01.932863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.932838 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.936807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.936786 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938173 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938182 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938220 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938234 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938241 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938238 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938228 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938261 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mzlcx\"" Nov 01 22:24:01.938279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938278 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 22:24:01.938635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.938322 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a6a3msr2p8too\"" Nov 01 22:24:01.942991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.942965 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 22:24:01.943102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.943031 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 22:24:01.943170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.943104 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 22:24:01.943170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.943157 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 22:24:01.943281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.943255 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 22:24:01.947770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.947755 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 22:24:01.950988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.950967 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:01.955358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.955323 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:01.967603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967565 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.967724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967618 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.967724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967638 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.967806 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967747 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.967839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967810 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.967839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967834 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.967930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967850 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.967930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.967866 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968039 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968088 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968139 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968187 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968222 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ft42\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-kube-api-access-8ft42\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968285 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968316 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968343 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968394 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968413 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968443 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:01.968590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:01.968541 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.069869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069787 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.069869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069831 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.069869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069850 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.069869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069867 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.070184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069886 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.070184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069914 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.070184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069933 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.070184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.069960 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.070513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070455 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070693 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ft42\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-kube-api-access-8ft42\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070737 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070772 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070795 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070826 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070854 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070865 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070884 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070910 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070940 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.070986 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.071016 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.071065 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.071476 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.071700 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.072849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.072491 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.073731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.072976 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.073731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.072976 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-web-config\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.073731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.073124 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.073731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.073643 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.073940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.073833 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.074849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.074275 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.074849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.074557 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.074849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.074804 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.075147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.075098 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.075782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.075764 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config-out\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.076093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.076073 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.076141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.076093 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.076185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.076135 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.076311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.076295 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.093438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.093412 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ft42\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-kube-api-access-8ft42\") pod \"prometheus-k8s-0\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.258254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.258214 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:02.553145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.551732 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:02.568851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.568827 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:24:02.569030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.569019 4981 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 22:24:02.600602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.600559 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-82z6g_4e15bc3a-dc26-4b68-bba9-b670d741c952/serve-healthcheck-canary/0.log" Nov 01 22:24:02.641917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.641879 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:24:02.647436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.647412 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2k4xp" Nov 01 22:24:02.866121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.866041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerStarted","Data":"18a94e3ecf5baf59ab0ce9aa96c4130950383fa79273629d8ffafa4a3760b289"} Nov 01 22:24:02.950759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:02.950727 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:24:03.146061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.145979 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:24:03.151270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.151246 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:24:03.872373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.872294 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" event={"ID":"3e7900c2-b948-46db-a1fa-535cf450e73b","Type":"ContainerStarted","Data":"2e1d0fa72a81a3fcbbcc646dc00c7ab68d923d55dcac73292c1ad6259b570671"} Nov 01 22:24:03.872373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.872366 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" event={"ID":"3e7900c2-b948-46db-a1fa-535cf450e73b","Type":"ContainerStarted","Data":"1518259277e77e2793984425ee49d7a1c5c2b61e78db986dd1b984551256e68e"} Nov 01 22:24:03.872907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.872408 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" event={"ID":"3e7900c2-b948-46db-a1fa-535cf450e73b","Type":"ContainerStarted","Data":"a354d1535f8bd2a58efe5322d242570443cfbd5faa3a831ad19235caf6021ee3"} Nov 01 22:24:03.877619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.877596 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:24:03.924730 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.924696 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d9849f88-4ntzt"] Nov 01 22:24:03.959212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:03.959184 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:04.048682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:04.048643 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:04.878507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:04.878408 4981 generic.go:356] "Generic (PLEG): container finished" podID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerID="6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd" exitCode=0 Nov 01 22:24:04.878507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:04.878476 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd"} Nov 01 22:24:04.881265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:04.881242 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" event={"ID":"c95c3ce6-d144-4423-94fc-65f47402f703","Type":"ContainerStarted","Data":"57334d93aac1de0125fd95decf9c538ac8caecac1e062b210bcafb94b58a72f9"} Nov 01 22:24:04.881331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:04.881296 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" event={"ID":"c95c3ce6-d144-4423-94fc-65f47402f703","Type":"ContainerStarted","Data":"df0a6560ad40e942d3d6f2f918bd847b97be71610d6d7b1a7f51d6aad7ae6a15"} Nov 01 22:24:04.881331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:04.881326 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" event={"ID":"c95c3ce6-d144-4423-94fc-65f47402f703","Type":"ContainerStarted","Data":"e68241e988c808340bd04c3c50b5d64d2fa449c2f04e55d955a33b006d453ebf"} Nov 01 22:24:04.940897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:04.940845 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69d449f8cd-dgrw9" podStartSLOduration=1.782789727 podStartE2EDuration="4.940830785s" totalImagesPullingTime="3.158041058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:04.938307057 +0000 UTC m=+10.366184985" watchObservedRunningTime="2026-11-01 22:24:04.940830785 +0000 UTC m=+10.368708713" Nov 01 22:24:05.591041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:05.590997 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:05.669121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:05.669090 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:05.889366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:05.889287 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" event={"ID":"3e7900c2-b948-46db-a1fa-535cf450e73b","Type":"ContainerStarted","Data":"d90ffe14a2814a59d24ec08d800df1fb7118fda119243c8e4b04ac21c975f3e4"} Nov 01 22:24:05.889366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:05.889348 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" event={"ID":"3e7900c2-b948-46db-a1fa-535cf450e73b","Type":"ContainerStarted","Data":"1f3d6910bad3d53f7de503291a70209e639152e691ffa667a7c8026d0db5b135"} Nov 01 22:24:05.889762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:05.889388 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" event={"ID":"3e7900c2-b948-46db-a1fa-535cf450e73b","Type":"ContainerStarted","Data":"14831e0561ae10ee0abac6c7921280c678f7a25d1d7083533a4b03c37a6cadcc"} Nov 01 22:24:05.915291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:05.915240 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" podStartSLOduration=2.436655801 podStartE2EDuration="8.915224136s" totalImagesPullingTime="6.478568335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:05.913662267 +0000 UTC m=+11.341540197" watchObservedRunningTime="2026-11-01 22:24:05.915224136 +0000 UTC m=+11.343102067" Nov 01 22:24:06.070651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:06.070610 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:06.892865 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:06.892833 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:24:08.379228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:08.379189 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:08.480567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:08.480517 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:09.362996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.362937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:09.463410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.463373 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:09.662525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.662490 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:09.906708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.906633 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerStarted","Data":"6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89"} Nov 01 22:24:09.906708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.906696 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerStarted","Data":"921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68"} Nov 01 22:24:09.906878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.906720 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerStarted","Data":"5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3"} Nov 01 22:24:09.906878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.906743 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerStarted","Data":"4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210"} Nov 01 22:24:09.906878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.906780 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerStarted","Data":"a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2"} Nov 01 22:24:09.906878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:09.906807 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerStarted","Data":"fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b"} Nov 01 22:24:10.193899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:10.193867 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:10.287772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:10.287742 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:10.520613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:10.520524 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2rvph" Nov 01 22:24:11.697859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:11.697803 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:11.800762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:11.800731 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:12.258556 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:12.258523 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:12.755970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:12.755930 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.507003952 podStartE2EDuration="11.755915744s" totalImagesPullingTime="6.231404236s" totalInitContainerRuntime="17.507556ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:09.97397274 +0000 UTC m=+15.401850669" watchObservedRunningTime="2026-11-01 22:24:12.755915744 +0000 UTC m=+18.183793838" Nov 01 22:24:12.901458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:12.901432 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6887556c4f-ppfhf" Nov 01 22:24:13.927039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:13.927004 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-n87cr" event={"ID":"8deca68d-1a0b-4841-9c91-0682262d59c7","Type":"ContainerStarted","Data":"94d21ac8cd0d7bc92c775bfd6dbf14e33c9328721eea1f0b4490f7cd26b1fdb7"} Nov 01 22:24:13.927468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:13.927315 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:24:13.928681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:13.928659 4981 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-n87cr container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.7:8080/\": dial tcp 10.134.0.7:8080: connect: connection refused" start-of-body= Nov 01 22:24:13.928779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:13.928705 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-n87cr" podUID="8deca68d-1a0b-4841-9c91-0682262d59c7" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.7:8080/\": dial tcp 10.134.0.7:8080: connect: connection refused" Nov 01 22:24:14.527055 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:14.527021 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:24:14.951096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:14.951068 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-n87cr" Nov 01 22:24:15.354001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:15.353925 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:15.389883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:15.389856 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:15.956480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:15.956450 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:17.984703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:17.984672 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dccbd4987-22x68"] Nov 01 22:24:18.497460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:18.497427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:18.599284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:18.599245 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:20.203628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:20.203527 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xglkp" Nov 01 22:24:20.601231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:20.601153 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:20.705750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:20.705639 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:25.182781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:25.182745 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:25.274263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:25.274230 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:26.344747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:26.344717 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:26.421980 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:26.421948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:28.948458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:28.948395 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-85d9849f88-4ntzt" podUID="2fe7ab41-3bca-400b-9a55-ed06fb29c216" containerName="console" containerID="cri-o://45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0" gracePeriod=15 Nov 01 22:24:29.215019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.214968 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:24:29.333333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333305 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\" (UniqueName: \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\") pod \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " Nov 01 22:24:29.333489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333371 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\") pod \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " Nov 01 22:24:29.333489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333400 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\") pod \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " Nov 01 22:24:29.333489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333427 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\") pod \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " Nov 01 22:24:29.333489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333455 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\") pod \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " Nov 01 22:24:29.333489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333483 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\") pod \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " Nov 01 22:24:29.333760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333516 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\") pod \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\" (UID: \"2fe7ab41-3bca-400b-9a55-ed06fb29c216\") " Nov 01 22:24:29.333827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333800 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert" pod "2fe7ab41-3bca-400b-9a55-ed06fb29c216" (UID: "2fe7ab41-3bca-400b-9a55-ed06fb29c216"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:29.334019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333976 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca" pod "2fe7ab41-3bca-400b-9a55-ed06fb29c216" (UID: "2fe7ab41-3bca-400b-9a55-ed06fb29c216"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:29.334141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333993 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp" pod "2fe7ab41-3bca-400b-9a55-ed06fb29c216" (UID: "2fe7ab41-3bca-400b-9a55-ed06fb29c216"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:29.334141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.333990 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config" pod "2fe7ab41-3bca-400b-9a55-ed06fb29c216" (UID: "2fe7ab41-3bca-400b-9a55-ed06fb29c216"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:29.335807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.335779 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config" pod "2fe7ab41-3bca-400b-9a55-ed06fb29c216" (UID: "2fe7ab41-3bca-400b-9a55-ed06fb29c216"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:29.335902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.335825 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk" pod "2fe7ab41-3bca-400b-9a55-ed06fb29c216" (UID: "2fe7ab41-3bca-400b-9a55-ed06fb29c216"). InnerVolumeSpecName "kube-api-access-l6gnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:29.335902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.335853 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert" pod "2fe7ab41-3bca-400b-9a55-ed06fb29c216" (UID: "2fe7ab41-3bca-400b-9a55-ed06fb29c216"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:29.434255 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.434222 4981 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-serving-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.434255 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.434255 4981 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.434438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.434270 4981 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-oauth-config\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.434438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.434285 4981 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fe7ab41-3bca-400b-9a55-ed06fb29c216-tmp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.434438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.434298 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6gnk\" (UniqueName: \"kubernetes.io/projected/2fe7ab41-3bca-400b-9a55-ed06fb29c216-kube-api-access-l6gnk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.434438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.434312 4981 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-oauth-serving-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.434438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.434327 4981 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fe7ab41-3bca-400b-9a55-ed06fb29c216-console-config\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:29.595428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.595355 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mpjqm" Nov 01 22:24:29.995877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.995843 4981 generic.go:356] "Generic (PLEG): container finished" podID="2fe7ab41-3bca-400b-9a55-ed06fb29c216" containerID="45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0" exitCode=0 Nov 01 22:24:29.996258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.995888 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d9849f88-4ntzt" Nov 01 22:24:29.996258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.995931 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d9849f88-4ntzt" event={"ID":"2fe7ab41-3bca-400b-9a55-ed06fb29c216","Type":"ContainerDied","Data":"45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0"} Nov 01 22:24:29.996258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.995981 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d9849f88-4ntzt" event={"ID":"2fe7ab41-3bca-400b-9a55-ed06fb29c216","Type":"ContainerDied","Data":"48fa1eb121bf952a08e72152a0367494f5dee58dac93cbc1725f9ac7a6a31454"} Nov 01 22:24:29.996258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:29.996013 4981 scope.go:122] "RemoveContainer" containerID="45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0" Nov 01 22:24:30.004390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:30.004374 4981 scope.go:122] "RemoveContainer" containerID="45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0" Nov 01 22:24:30.004660 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:30.004639 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0\": container with ID starting with 45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0 not found: ID does not exist" containerID="45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0" Nov 01 22:24:30.004714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:30.004669 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0"} err="failed to get container status \"45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0\": rpc error: code = NotFound desc = could not find container \"45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0\": container with ID starting with 45f6bd0578e9ca1240d8bef859f0b95dbda237558478208436fbcfacf7f083e0 not found: ID does not exist" Nov 01 22:24:30.016731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:30.016709 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85d9849f88-4ntzt"] Nov 01 22:24:30.020413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:30.020393 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85d9849f88-4ntzt"] Nov 01 22:24:30.371292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:30.371207 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:30.457963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:30.457927 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:30.731348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:30.731317 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe7ab41-3bca-400b-9a55-ed06fb29c216" path="/var/lib/kubelet/pods/2fe7ab41-3bca-400b-9a55-ed06fb29c216/volumes" Nov 01 22:24:35.582831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:35.582799 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:35.653052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:35.653018 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:36.161305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:36.161267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:36.261028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:36.260989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:40.625097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:40.625063 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:40.700118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:40.700085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:42.949987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:42.949944 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:43.007009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.006975 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7dccbd4987-22x68" podUID="c2beea1c-4d22-435c-8214-283eca213a15" containerName="console" containerID="cri-o://fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c" gracePeriod=15 Nov 01 22:24:43.029895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.029866 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:43.261952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.261928 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:24:43.359867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.359841 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-serving-cert\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.359883 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2beea1c-4d22-435c-8214-283eca213a15-kube-api-access-h486v\" (UniqueName: \"kubernetes.io/projected/c2beea1c-4d22-435c-8214-283eca213a15-kube-api-access-h486v\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.359914 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2beea1c-4d22-435c-8214-283eca213a15-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2beea1c-4d22-435c-8214-283eca213a15-tmp\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360034 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-oauth-config\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360076 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-service-ca\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360117 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-oauth-serving-cert\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360151 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2beea1c-4d22-435c-8214-283eca213a15-tmp" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:43.360287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360230 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-trusted-ca-bundle\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360269 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-console-config\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-console-config\") pod \"c2beea1c-4d22-435c-8214-283eca213a15\" (UID: \"c2beea1c-4d22-435c-8214-283eca213a15\") " Nov 01 22:24:43.360534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360468 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-service-ca" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:43.360534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360491 4981 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2beea1c-4d22-435c-8214-283eca213a15-tmp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.360699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360560 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-trusted-ca-bundle" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:43.360699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360568 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-oauth-serving-cert" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:43.360870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.360846 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-console-config" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:43.362073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.362052 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-serving-cert" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:43.362186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.362109 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-oauth-config" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:43.362242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.362193 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2beea1c-4d22-435c-8214-283eca213a15-kube-api-access-h486v" pod "c2beea1c-4d22-435c-8214-283eca213a15" (UID: "c2beea1c-4d22-435c-8214-283eca213a15"). InnerVolumeSpecName "kube-api-access-h486v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:43.461260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.461233 4981 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-trusted-ca-bundle\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.461260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.461254 4981 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-console-config\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.461260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.461264 4981 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-serving-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.461447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.461273 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h486v\" (UniqueName: \"kubernetes.io/projected/c2beea1c-4d22-435c-8214-283eca213a15-kube-api-access-h486v\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.461447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.461283 4981 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c2beea1c-4d22-435c-8214-283eca213a15-console-oauth-config\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.461447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.461291 4981 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.461447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.461299 4981 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c2beea1c-4d22-435c-8214-283eca213a15-oauth-serving-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:43.520949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:43.520883 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:44.048168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.048135 4981 generic.go:356] "Generic (PLEG): container finished" podID="c2beea1c-4d22-435c-8214-283eca213a15" containerID="fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c" exitCode=0 Nov 01 22:24:44.048643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.048186 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dccbd4987-22x68" Nov 01 22:24:44.048643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.048193 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccbd4987-22x68" event={"ID":"c2beea1c-4d22-435c-8214-283eca213a15","Type":"ContainerDied","Data":"fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c"} Nov 01 22:24:44.048643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.048231 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dccbd4987-22x68" event={"ID":"c2beea1c-4d22-435c-8214-283eca213a15","Type":"ContainerDied","Data":"661197cba491feebe1d16441babbc6e79cc66ca6674221df58b4265e8d7476ca"} Nov 01 22:24:44.048643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.048249 4981 scope.go:122] "RemoveContainer" containerID="fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c" Nov 01 22:24:44.057272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.057251 4981 scope.go:122] "RemoveContainer" containerID="fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c" Nov 01 22:24:44.057519 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:44.057497 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c\": container with ID starting with fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c not found: ID does not exist" containerID="fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c" Nov 01 22:24:44.057608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.057530 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c"} err="failed to get container status \"fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c\": rpc error: code = NotFound desc = could not find container \"fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c\": container with ID starting with fce9f8fdc28e2168aebc188ee5ef4d1841ae18ac798425bb2e6c44ae3fafe09c not found: ID does not exist" Nov 01 22:24:44.072485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.072456 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dccbd4987-22x68"] Nov 01 22:24:44.076451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.076430 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dccbd4987-22x68"] Nov 01 22:24:44.521711 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:44.521674 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:24:44.619431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.619400 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:44.619784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.619758 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="prometheus" containerID="cri-o://fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b" gracePeriod=600 Nov 01 22:24:44.619881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.619772 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="config-reloader" containerID="cri-o://a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2" gracePeriod=600 Nov 01 22:24:44.619881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.619836 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-web" containerID="cri-o://5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3" gracePeriod=600 Nov 01 22:24:44.619881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.619771 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="thanos-sidecar" containerID="cri-o://4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210" gracePeriod=600 Nov 01 22:24:44.620029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.619916 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89" gracePeriod=600 Nov 01 22:24:44.620029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.619937 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy" containerID="cri-o://921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68" gracePeriod=600 Nov 01 22:24:44.732538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:44.732516 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2beea1c-4d22-435c-8214-283eca213a15" path="/var/lib/kubelet/pods/c2beea1c-4d22-435c-8214-283eca213a15/volumes" Nov 01 22:24:45.055521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055494 4981 generic.go:356] "Generic (PLEG): container finished" podID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerID="6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89" exitCode=0 Nov 01 22:24:45.055521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055515 4981 generic.go:356] "Generic (PLEG): container finished" podID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerID="921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68" exitCode=0 Nov 01 22:24:45.055521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055522 4981 generic.go:356] "Generic (PLEG): container finished" podID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerID="4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210" exitCode=0 Nov 01 22:24:45.055521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055530 4981 generic.go:356] "Generic (PLEG): container finished" podID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerID="a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2" exitCode=0 Nov 01 22:24:45.055985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055535 4981 generic.go:356] "Generic (PLEG): container finished" podID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerID="fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b" exitCode=0 Nov 01 22:24:45.055985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055554 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89"} Nov 01 22:24:45.055985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055615 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68"} Nov 01 22:24:45.055985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055640 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210"} Nov 01 22:24:45.055985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055665 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2"} Nov 01 22:24:45.055985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.055688 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b"} Nov 01 22:24:45.854502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.854479 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:45.980639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980553 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.980639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980608 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-grpc-tls\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.980639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980629 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-metrics-client-ca\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.980869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980654 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-kube-api-access-8ft42\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-kube-api-access-8ft42\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.980869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980763 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.980869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980803 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-kubelet-serving-ca-bundle\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.980869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980845 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config-out\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980881 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-2\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980909 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-serving-certs-ca-bundle\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980945 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-kube-rbac-proxy\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.980981 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981007 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-1\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981055 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-0\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981109 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-thanos-prometheus-http-client-file\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981109 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-metrics-client-ca" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981152 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-tls\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981181 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-trusted-ca-bundle\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981215 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-web-config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-web-config\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981245 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-db\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981255 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-kubelet-serving-ca-bundle" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981276 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-tls-assets\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-tls-assets\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981309 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-metrics-client-certs\") pod \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\" (UID: \"fd27e5e1-ba25-4802-827e-3f982f1b32fa\") " Nov 01 22:24:45.981850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981558 4981 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-metrics-client-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:45.981850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981607 4981 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:45.982022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.981997 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-2" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:45.982107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.982032 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-trusted-ca-bundle" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:45.982373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.982346 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-serving-certs-ca-bundle" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:45.982456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.982375 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-1" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:45.983768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.983739 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-kube-api-access-8ft42" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "kube-api-access-8ft42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:45.983859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.983814 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-grpc-tls" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.983910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.983868 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.983970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.983949 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-db" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:45.984113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.984087 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-0" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:24:45.984838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.984809 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config-out" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:24:45.985054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.985032 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.985465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.985430 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-tls" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.985778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.985754 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-thanos-sidecar-tls" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.985879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.985782 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-metrics-client-certs" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.986066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.986047 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-tls-assets" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:24:45.986167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.986151 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-kube-rbac-proxy" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.986790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.986771 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-thanos-prometheus-http-client-file" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:45.995058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:45.994984 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-web-config" pod "fd27e5e1-ba25-4802-827e-3f982f1b32fa" (UID: "fd27e5e1-ba25-4802-827e-3f982f1b32fa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:24:46.062256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.062221 4981 generic.go:356] "Generic (PLEG): container finished" podID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerID="5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3" exitCode=0 Nov 01 22:24:46.062661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.062294 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.062661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.062304 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3"} Nov 01 22:24:46.062661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.062371 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fd27e5e1-ba25-4802-827e-3f982f1b32fa","Type":"ContainerDied","Data":"18a94e3ecf5baf59ab0ce9aa96c4130950383fa79273629d8ffafa4a3760b289"} Nov 01 22:24:46.062661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.062405 4981 scope.go:122] "RemoveContainer" containerID="6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89" Nov 01 22:24:46.070681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.070666 4981 scope.go:122] "RemoveContainer" containerID="921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68" Nov 01 22:24:46.077698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.077680 4981 scope.go:122] "RemoveContainer" containerID="5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3" Nov 01 22:24:46.084908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084785 4981 scope.go:122] "RemoveContainer" containerID="4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210" Nov 01 22:24:46.084908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084880 4981 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.084908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084903 4981 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084918 4981 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084933 4981 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084953 4981 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084966 4981 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.084998 4981 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-web-config\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085012 4981 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-db\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085029 4981 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-tls-assets\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085042 4981 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-metrics-client-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085056 4981 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085070 4981 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-grpc-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085087 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ft42\" (UniqueName: \"kubernetes.io/projected/fd27e5e1-ba25-4802-827e-3f982f1b32fa-kube-api-access-8ft42\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085100 4981 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085113 4981 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd27e5e1-ba25-4802-827e-3f982f1b32fa-config-out\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085130 4981 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085143 4981 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd27e5e1-ba25-4802-827e-3f982f1b32fa-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.085393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.085156 4981 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd27e5e1-ba25-4802-827e-3f982f1b32fa-secret-kube-rbac-proxy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:24:46.088371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.088349 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:46.091560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.091540 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:46.094445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.094429 4981 scope.go:122] "RemoveContainer" containerID="a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2" Nov 01 22:24:46.100626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.100605 4981 scope.go:122] "RemoveContainer" containerID="fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b" Nov 01 22:24:46.107745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.107728 4981 scope.go:122] "RemoveContainer" containerID="6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd" Nov 01 22:24:46.114118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.114102 4981 scope.go:122] "RemoveContainer" containerID="6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89" Nov 01 22:24:46.114358 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:46.114337 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89\": container with ID starting with 6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89 not found: ID does not exist" containerID="6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89" Nov 01 22:24:46.114441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.114364 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89"} err="failed to get container status \"6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89\": rpc error: code = NotFound desc = could not find container \"6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89\": container with ID starting with 6c9771f52697b37ba4fe0295bf636954a8991414c4c409a9b339353adfe4fa89 not found: ID does not exist" Nov 01 22:24:46.114441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.114382 4981 scope.go:122] "RemoveContainer" containerID="921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68" Nov 01 22:24:46.114614 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:46.114596 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68\": container with ID starting with 921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68 not found: ID does not exist" containerID="921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68" Nov 01 22:24:46.114657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.114618 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68"} err="failed to get container status \"921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68\": rpc error: code = NotFound desc = could not find container \"921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68\": container with ID starting with 921c484cac94b3d7d5f2921d1a45a8b3773c11872a937f623cb16295a2cdbe68 not found: ID does not exist" Nov 01 22:24:46.114657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.114631 4981 scope.go:122] "RemoveContainer" containerID="5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3" Nov 01 22:24:46.114868 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:46.114850 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3\": container with ID starting with 5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3 not found: ID does not exist" containerID="5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3" Nov 01 22:24:46.114923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.114873 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3"} err="failed to get container status \"5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3\": rpc error: code = NotFound desc = could not find container \"5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3\": container with ID starting with 5c0eb3a66d1fa9ae0e0aa4953171377c90d62ae56b64e956b6b88d15dce06ba3 not found: ID does not exist" Nov 01 22:24:46.114923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.114889 4981 scope.go:122] "RemoveContainer" containerID="4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210" Nov 01 22:24:46.115101 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:46.115086 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210\": container with ID starting with 4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210 not found: ID does not exist" containerID="4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210" Nov 01 22:24:46.115138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.115105 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210"} err="failed to get container status \"4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210\": rpc error: code = NotFound desc = could not find container \"4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210\": container with ID starting with 4ae917ae25339183948de02bc749080254c783eabc133ee91d463456c9b46210 not found: ID does not exist" Nov 01 22:24:46.115138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.115116 4981 scope.go:122] "RemoveContainer" containerID="a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2" Nov 01 22:24:46.115306 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:46.115290 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2\": container with ID starting with a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2 not found: ID does not exist" containerID="a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2" Nov 01 22:24:46.115341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.115311 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2"} err="failed to get container status \"a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2\": rpc error: code = NotFound desc = could not find container \"a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2\": container with ID starting with a58fe4d99b9a6900218941684233d4213805b64597d16cb8b127713bd5caaeb2 not found: ID does not exist" Nov 01 22:24:46.115341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.115324 4981 scope.go:122] "RemoveContainer" containerID="fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b" Nov 01 22:24:46.115549 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:46.115534 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b\": container with ID starting with fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b not found: ID does not exist" containerID="fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b" Nov 01 22:24:46.115604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.115552 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b"} err="failed to get container status \"fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b\": rpc error: code = NotFound desc = could not find container \"fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b\": container with ID starting with fd757f2220da0e2aa862a8f5b32012ca9b5373d2fc2a6ff04694f93582ce9e4b not found: ID does not exist" Nov 01 22:24:46.115604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.115564 4981 scope.go:122] "RemoveContainer" containerID="6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd" Nov 01 22:24:46.115795 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:24:46.115774 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd\": container with ID starting with 6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd not found: ID does not exist" containerID="6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd" Nov 01 22:24:46.115869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.115800 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd"} err="failed to get container status \"6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd\": rpc error: code = NotFound desc = could not find container \"6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd\": container with ID starting with 6758543d348453941c14e6bd875d0eda0b2b81477d08614f5ab922750468f4dd not found: ID does not exist" Nov 01 22:24:46.117848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.117830 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:46.118155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118141 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c2beea1c-4d22-435c-8214-283eca213a15" containerName="console" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118158 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2beea1c-4d22-435c-8214-283eca213a15" containerName="console" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118173 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="prometheus" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118180 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="prometheus" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118194 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="thanos-sidecar" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118201 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="thanos-sidecar" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118211 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2fe7ab41-3bca-400b-9a55-ed06fb29c216" containerName="console" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118218 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe7ab41-3bca-400b-9a55-ed06fb29c216" containerName="console" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118227 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="config-reloader" Nov 01 22:24:46.118233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118233 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="config-reloader" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118252 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-web" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118260 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-web" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118269 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-thanos" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118276 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-thanos" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118290 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="init-config-reloader" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118296 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="init-config-reloader" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118309 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118316 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118375 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="thanos-sidecar" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118387 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe7ab41-3bca-400b-9a55-ed06fb29c216" containerName="console" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118397 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="config-reloader" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118408 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-web" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118416 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy-thanos" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118426 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="kube-rbac-proxy" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118439 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2beea1c-4d22-435c-8214-283eca213a15" containerName="console" Nov 01 22:24:46.118695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.118447 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" containerName="prometheus" Nov 01 22:24:46.122101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.122084 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.129214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129195 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 22:24:46.129214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129209 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 22:24:46.129406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129244 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 22:24:46.129692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129676 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 22:24:46.129767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129702 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mzlcx\"" Nov 01 22:24:46.129767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129708 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 22:24:46.129767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129719 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 22:24:46.129925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129813 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 22:24:46.129925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129843 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 22:24:46.129925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129846 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 22:24:46.129925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129850 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 22:24:46.129925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129892 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 22:24:46.129925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129903 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a6a3msr2p8too\"" Nov 01 22:24:46.129925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.129909 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 22:24:46.131293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.131280 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 22:24:46.137228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.137210 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 22:24:46.137315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.137277 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 22:24:46.151666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.151646 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:46.186240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186213 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186247 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49fa04e-24ec-496c-958b-c268bf039b7e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186276 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186401 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186448 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186503 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186532 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186547 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhhx\" (UniqueName: \"kubernetes.io/projected/e49fa04e-24ec-496c-958b-c268bf039b7e-kube-api-access-pkhhx\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186570 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186627 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186666 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186701 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186732 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186782 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49fa04e-24ec-496c-958b-c268bf039b7e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186802 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-config\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186842 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186867 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186894 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.186926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186911 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.187094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.186934 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287207 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287237 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287258 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287272 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287288 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287301 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhhx\" (UniqueName: \"kubernetes.io/projected/e49fa04e-24ec-496c-958b-c268bf039b7e-kube-api-access-pkhhx\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287327 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287629 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287664 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.287787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287708 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287977 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.287712 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288262 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288359 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288335 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288316 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288465 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49fa04e-24ec-496c-958b-c268bf039b7e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288503 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-config\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288526 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288551 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288606 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288635 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288670 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288702 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.288769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288731 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49fa04e-24ec-496c-958b-c268bf039b7e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.289006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.288859 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.290115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.290085 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.290966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.290420 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.290966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.290475 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.290966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.290928 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.291490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.291465 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.292010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.291940 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e49fa04e-24ec-496c-958b-c268bf039b7e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.292164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.292130 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e49fa04e-24ec-496c-958b-c268bf039b7e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.292339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.292319 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-config\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.292402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.292324 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.292489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.292471 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.293001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.292981 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e49fa04e-24ec-496c-958b-c268bf039b7e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.293182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.293167 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.293559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.293544 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e49fa04e-24ec-496c-958b-c268bf039b7e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.295850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.295833 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhhx\" (UniqueName: \"kubernetes.io/projected/e49fa04e-24ec-496c-958b-c268bf039b7e-kube-api-access-pkhhx\") pod \"prometheus-k8s-0\" (UID: \"e49fa04e-24ec-496c-958b-c268bf039b7e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.437934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.437902 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:46.714207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.714183 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 22:24:46.715699 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:24:46.715674 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49fa04e_24ec_496c_958b_c268bf039b7e.slice/crio-6eb68da92137dbd39d718186f7a89c7cfc10f47c5863620f0ce3841000af73b3 WatchSource:0}: Error finding container 6eb68da92137dbd39d718186f7a89c7cfc10f47c5863620f0ce3841000af73b3: Status 404 returned error can't find the container with id 6eb68da92137dbd39d718186f7a89c7cfc10f47c5863620f0ce3841000af73b3 Nov 01 22:24:46.732402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:46.732372 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd27e5e1-ba25-4802-827e-3f982f1b32fa" path="/var/lib/kubelet/pods/fd27e5e1-ba25-4802-827e-3f982f1b32fa/volumes" Nov 01 22:24:47.068381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:47.068299 4981 generic.go:356] "Generic (PLEG): container finished" podID="e49fa04e-24ec-496c-958b-c268bf039b7e" containerID="c2bfaa69f53467e5f0bdf783ef1ee98d6d48ec7af0b6c81360862f852f8dfd9d" exitCode=0 Nov 01 22:24:47.068758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:47.068383 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerDied","Data":"c2bfaa69f53467e5f0bdf783ef1ee98d6d48ec7af0b6c81360862f852f8dfd9d"} Nov 01 22:24:47.068758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:47.068450 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerStarted","Data":"6eb68da92137dbd39d718186f7a89c7cfc10f47c5863620f0ce3841000af73b3"} Nov 01 22:24:48.075066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:48.075031 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerStarted","Data":"ef78b504e09bcd29586206e00f24ce6ab4cc739fe995c7474d6d0b6e70e87acc"} Nov 01 22:24:48.075544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:48.075115 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerStarted","Data":"f865a78ed8845bf60dd7a29c25ac4bf4f7f024db765e24d4c4c2c29c09df398c"} Nov 01 22:24:48.075544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:48.075145 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerStarted","Data":"abbddc9f4e23bcfe5d463120e05db57288f106224a46ef1c1b6960614a241461"} Nov 01 22:24:48.075544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:48.075171 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerStarted","Data":"31570535f334afae8f7da55eca7a9ad5beee2e398bde7b82d2fa34c76663aef0"} Nov 01 22:24:48.075544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:48.075194 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerStarted","Data":"b310804d48e0ca466260a820260ea8e1c8616b3a4281936b9b858cffd478170a"} Nov 01 22:24:48.075544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:48.075215 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e49fa04e-24ec-496c-958b-c268bf039b7e","Type":"ContainerStarted","Data":"acbe8a11d53849bce4907bf45820f01197d1561b63ce143e4e4d2c280a803443"} Nov 01 22:24:48.103696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:48.103645 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.088810698 podStartE2EDuration="2.103631668s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.82097ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:48.101273084 +0000 UTC m=+53.529151016" watchObservedRunningTime="2026-11-01 22:24:48.103631668 +0000 UTC m=+53.531509597" Nov 01 22:24:50.588143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:50.588110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:50.660465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:50.660439 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:51.438688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:51.438650 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:24:56.454876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:56.454840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:56.553512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:56.553460 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:24:56.625012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:24:56.624986 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:11.737886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:11.737847 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:14.521099 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:25:14.521063 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:25:28.587287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:28.587254 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:28.664993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:28.664962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:40.000979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:40.000882 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:25:40.016693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:40.016661 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:25:40.267469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:40.267392 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 22:25:44.523045 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:25:44.523013 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:25:57.127403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:57.127372 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:25:57.201609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:25:57.201562 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:11.792749 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:26:11.792716 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:26:14.522897 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:26:14.522855 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:26:44.522715 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:26:44.522681 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:27:06.292569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:06.292495 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:09.171465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:09.171433 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:09.239756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:09.239729 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:11.826954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:11.826917 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:14.522258 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:27:14.522223 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:27:29.206545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:29.206510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:33.407550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:33.407516 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:33.507092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:33.507057 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:33.608702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:27:33.608672 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:27:44.522712 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:27:44.522681 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:28:11.872184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:28:11.872148 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:14.522800 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:28:14.522762 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:28:40.257756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:28:40.257687 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:28:44.521854 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:28:44.521817 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:29:08.361999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:08.361962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:08.651686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:08.651608 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:08.786414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:08.786385 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:10.110847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:10.110814 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:10.243351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:10.243313 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:11.246200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:11.246160 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:11.917446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:11.917419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:14.522768 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:29:14.522731 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:29:15.444422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:15.444379 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:15.544934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:15.544903 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:15.781720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:15.781645 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:15.875694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:15.875669 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:15.975987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:15.975960 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:18.349181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:18.349153 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:44.522662 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:29:44.522622 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:29:49.678037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:49.678001 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:50.448017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:50.447981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:53.210635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:53.210596 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:53.347342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:53.347297 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:55.397776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.397741 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:55.592854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.592818 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc"] Nov 01 22:29:55.596067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.596051 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:55.602373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.602346 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:55.605724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.605704 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-n6lsl\"" Nov 01 22:29:55.606958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.606943 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 22:29:55.607019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.606963 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 22:29:55.612369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.612350 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b224070e-06e8-4c40-afdf-2aa6765aae2f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-r44kc\" (UID: \"b224070e-06e8-4c40-afdf-2aa6765aae2f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:55.612453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.612393 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9849g\" (UniqueName: \"kubernetes.io/projected/b224070e-06e8-4c40-afdf-2aa6765aae2f-kube-api-access-9849g\") pod \"cert-manager-webhook-5dbbbf75cc-r44kc\" (UID: \"b224070e-06e8-4c40-afdf-2aa6765aae2f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:55.616517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.616496 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc"] Nov 01 22:29:55.712827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.712793 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b224070e-06e8-4c40-afdf-2aa6765aae2f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-r44kc\" (UID: \"b224070e-06e8-4c40-afdf-2aa6765aae2f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:55.712984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.712848 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9849g\" (UniqueName: \"kubernetes.io/projected/b224070e-06e8-4c40-afdf-2aa6765aae2f-kube-api-access-9849g\") pod \"cert-manager-webhook-5dbbbf75cc-r44kc\" (UID: \"b224070e-06e8-4c40-afdf-2aa6765aae2f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:55.738419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.738394 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9849g\" (UniqueName: \"kubernetes.io/projected/b224070e-06e8-4c40-afdf-2aa6765aae2f-kube-api-access-9849g\") pod \"cert-manager-webhook-5dbbbf75cc-r44kc\" (UID: \"b224070e-06e8-4c40-afdf-2aa6765aae2f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:55.740990 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.740964 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b224070e-06e8-4c40-afdf-2aa6765aae2f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-r44kc\" (UID: \"b224070e-06e8-4c40-afdf-2aa6765aae2f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:55.924426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:55.924392 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:29:56.234527 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:56.234500 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc"] Nov 01 22:29:56.236011 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:29:56.235981 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb224070e_06e8_4c40_afdf_2aa6765aae2f.slice/crio-2fdc4d4b3e2d22843e051b48964e2cbfb1fdcd7443720f139557bbe674a6c56b WatchSource:0}: Error finding container 2fdc4d4b3e2d22843e051b48964e2cbfb1fdcd7443720f139557bbe674a6c56b: Status 404 returned error can't find the container with id 2fdc4d4b3e2d22843e051b48964e2cbfb1fdcd7443720f139557bbe674a6c56b Nov 01 22:29:56.238780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:56.238125 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:29:57.135851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:57.135814 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" event={"ID":"b224070e-06e8-4c40-afdf-2aa6765aae2f","Type":"ContainerStarted","Data":"2fdc4d4b3e2d22843e051b48964e2cbfb1fdcd7443720f139557bbe674a6c56b"} Nov 01 22:29:57.820091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:57.820061 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:58.057318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:58.057288 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:29:58.155512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:29:58.155440 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:00.207065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:00.207033 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:00.533425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:00.533354 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:00.634091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:00.634062 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:02.157262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.157228 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf"] Nov 01 22:30:02.168961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.168929 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.171266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.171206 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf"] Nov 01 22:30:02.179659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.179633 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8sxpn\"" Nov 01 22:30:02.233877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.233837 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:02.269020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.268983 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2127ef0-ecd1-4bca-bc05-d84dc25c1274-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rtncf\" (UID: \"d2127ef0-ecd1-4bca-bc05-d84dc25c1274\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.269204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.269043 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxcj2\" (UniqueName: \"kubernetes.io/projected/d2127ef0-ecd1-4bca-bc05-d84dc25c1274-kube-api-access-xxcj2\") pod \"cert-manager-cainjector-5c57bbd4fc-rtncf\" (UID: \"d2127ef0-ecd1-4bca-bc05-d84dc25c1274\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.335111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.335079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:02.370373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.370324 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2127ef0-ecd1-4bca-bc05-d84dc25c1274-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rtncf\" (UID: \"d2127ef0-ecd1-4bca-bc05-d84dc25c1274\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.370555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.370401 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxcj2\" (UniqueName: \"kubernetes.io/projected/d2127ef0-ecd1-4bca-bc05-d84dc25c1274-kube-api-access-xxcj2\") pod \"cert-manager-cainjector-5c57bbd4fc-rtncf\" (UID: \"d2127ef0-ecd1-4bca-bc05-d84dc25c1274\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.410127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.410058 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxcj2\" (UniqueName: \"kubernetes.io/projected/d2127ef0-ecd1-4bca-bc05-d84dc25c1274-kube-api-access-xxcj2\") pod \"cert-manager-cainjector-5c57bbd4fc-rtncf\" (UID: \"d2127ef0-ecd1-4bca-bc05-d84dc25c1274\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.416702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.416668 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2127ef0-ecd1-4bca-bc05-d84dc25c1274-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-rtncf\" (UID: \"d2127ef0-ecd1-4bca-bc05-d84dc25c1274\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.484821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.484780 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" Nov 01 22:30:02.764429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.764397 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf"] Nov 01 22:30:02.768270 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:30:02.768234 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2127ef0_ecd1_4bca_bc05_d84dc25c1274.slice/crio-963b12b9c0ccf056847eb3bfdcba1e60bac95f3f16059e6be4e8ca51d65b13ca WatchSource:0}: Error finding container 963b12b9c0ccf056847eb3bfdcba1e60bac95f3f16059e6be4e8ca51d65b13ca: Status 404 returned error can't find the container with id 963b12b9c0ccf056847eb3bfdcba1e60bac95f3f16059e6be4e8ca51d65b13ca Nov 01 22:30:02.785866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:02.785832 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:03.161150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:03.161070 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" event={"ID":"d2127ef0-ecd1-4bca-bc05-d84dc25c1274","Type":"ContainerStarted","Data":"963b12b9c0ccf056847eb3bfdcba1e60bac95f3f16059e6be4e8ca51d65b13ca"} Nov 01 22:30:03.574544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:03.574513 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:04.171168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.171132 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" event={"ID":"b224070e-06e8-4c40-afdf-2aa6765aae2f","Type":"ContainerStarted","Data":"892897ebfcce9e3eda5e99e23ae47d88c350e721402440cf56afe4414f8cafe2"} Nov 01 22:30:04.171571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.171198 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:30:04.192522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.192468 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podStartSLOduration=2.138155929 podStartE2EDuration="9.192454628s" totalImagesPullingTime="7.054298699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:04.191377447 +0000 UTC m=+369.619255375" watchObservedRunningTime="2026-11-01 22:30:04.192454628 +0000 UTC m=+369.620332556" Nov 01 22:30:04.376292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.376253 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:04.969740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.969659 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw"] Nov 01 22:30:04.972955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.972938 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:04.984301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.984273 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 22:30:04.984301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.984295 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 22:30:04.984512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.984386 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-n74wd\"" Nov 01 22:30:04.985460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:04.985430 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw"] Nov 01 22:30:05.092735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.092695 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74dc6f3c-838b-4ea7-9b81-3f0890e8955b-tmp\") pod \"jobset-operator-674cd59dc5-7l5cw\" (UID: \"74dc6f3c-838b-4ea7-9b81-3f0890e8955b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:05.092735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.092740 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjm2\" (UniqueName: \"kubernetes.io/projected/74dc6f3c-838b-4ea7-9b81-3f0890e8955b-kube-api-access-ptjm2\") pod \"jobset-operator-674cd59dc5-7l5cw\" (UID: \"74dc6f3c-838b-4ea7-9b81-3f0890e8955b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:05.194091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.194055 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74dc6f3c-838b-4ea7-9b81-3f0890e8955b-tmp\") pod \"jobset-operator-674cd59dc5-7l5cw\" (UID: \"74dc6f3c-838b-4ea7-9b81-3f0890e8955b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:05.195784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.194102 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjm2\" (UniqueName: \"kubernetes.io/projected/74dc6f3c-838b-4ea7-9b81-3f0890e8955b-kube-api-access-ptjm2\") pod \"jobset-operator-674cd59dc5-7l5cw\" (UID: \"74dc6f3c-838b-4ea7-9b81-3f0890e8955b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:05.195784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.194493 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74dc6f3c-838b-4ea7-9b81-3f0890e8955b-tmp\") pod \"jobset-operator-674cd59dc5-7l5cw\" (UID: \"74dc6f3c-838b-4ea7-9b81-3f0890e8955b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:05.213159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.213123 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjm2\" (UniqueName: \"kubernetes.io/projected/74dc6f3c-838b-4ea7-9b81-3f0890e8955b-kube-api-access-ptjm2\") pod \"jobset-operator-674cd59dc5-7l5cw\" (UID: \"74dc6f3c-838b-4ea7-9b81-3f0890e8955b\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:05.284547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.284516 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" Nov 01 22:30:05.567911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.567832 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw"] Nov 01 22:30:05.570848 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:30:05.570800 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74dc6f3c_838b_4ea7_9b81_3f0890e8955b.slice/crio-c8a6a85105297a2311c6191c788c107dea896762b70e6ccbfea121d476e0d089 WatchSource:0}: Error finding container c8a6a85105297a2311c6191c788c107dea896762b70e6ccbfea121d476e0d089: Status 404 returned error can't find the container with id c8a6a85105297a2311c6191c788c107dea896762b70e6ccbfea121d476e0d089 Nov 01 22:30:05.599291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:05.599237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:06.166552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:06.166521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:06.179762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:06.179722 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" event={"ID":"74dc6f3c-838b-4ea7-9b81-3f0890e8955b","Type":"ContainerStarted","Data":"c8a6a85105297a2311c6191c788c107dea896762b70e6ccbfea121d476e0d089"} Nov 01 22:30:06.181136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:06.181116 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" event={"ID":"d2127ef0-ecd1-4bca-bc05-d84dc25c1274","Type":"ContainerStarted","Data":"2664c025723028cc80f62e2c1b55c4ed57ba6b7f3d72160e8dc0e35304c0bd0f"} Nov 01 22:30:06.210472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:06.210423 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-rtncf" podStartSLOduration=1.739390109 podStartE2EDuration="4.210409942s" totalImagesPullingTime="2.471019833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:06.209594724 +0000 UTC m=+371.637472647" watchObservedRunningTime="2026-11-01 22:30:06.210409942 +0000 UTC m=+371.638287873" Nov 01 22:30:06.285013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:06.284981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:07.790141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:07.790105 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:07.891905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:07.891865 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:08.916873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:08.916839 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.072861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.072806 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8"] Nov 01 22:30:09.076388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.076364 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.115608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.115538 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8"] Nov 01 22:30:09.130866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.130822 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.131087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.130879 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2af45373-36c6-4d05-93c1-1a0a1805e7c2-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.131087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.130998 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7jl\" (UniqueName: \"kubernetes.io/projected/2af45373-36c6-4d05-93c1-1a0a1805e7c2-kube-api-access-7q7jl\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.131087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.131038 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.131087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.131062 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2af45373-36c6-4d05-93c1-1a0a1805e7c2-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.131290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.131159 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.160603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.160544 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 22:30:09.160603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.160593 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 22:30:09.160603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.160604 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bqmh4\"" Nov 01 22:30:09.160846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.160556 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 22:30:09.160846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.160597 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 22:30:09.160846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.160679 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 22:30:09.207374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.207337 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.232552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.232506 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7jl\" (UniqueName: \"kubernetes.io/projected/2af45373-36c6-4d05-93c1-1a0a1805e7c2-kube-api-access-7q7jl\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.232762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.232606 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.232762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.232643 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2af45373-36c6-4d05-93c1-1a0a1805e7c2-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.232762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.232677 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.232762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.232739 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.232989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.232768 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2af45373-36c6-4d05-93c1-1a0a1805e7c2-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.233548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.233513 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2af45373-36c6-4d05-93c1-1a0a1805e7c2-manifests\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.235340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.235319 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.235617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.235568 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.235833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.235813 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2af45373-36c6-4d05-93c1-1a0a1805e7c2-tmp\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.236020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.236002 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2af45373-36c6-4d05-93c1-1a0a1805e7c2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.241348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.241325 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7jl\" (UniqueName: \"kubernetes.io/projected/2af45373-36c6-4d05-93c1-1a0a1805e7c2-kube-api-access-7q7jl\") pod \"opendatahub-operator-controller-manager-5b59bbd84f-k42k8\" (UID: \"2af45373-36c6-4d05-93c1-1a0a1805e7c2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.314892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.314857 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.395089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.395050 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:09.408882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.408854 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.534293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.534196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.635910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.635878 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:09.707606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:09.706914 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8"] Nov 01 22:30:09.716078 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:30:09.716039 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af45373_36c6_4d05_93c1_1a0a1805e7c2.slice/crio-54419ec6348582247bf76d9d66987e86b48d93618de590b18c0d8630d9dbef2d WatchSource:0}: Error finding container 54419ec6348582247bf76d9d66987e86b48d93618de590b18c0d8630d9dbef2d: Status 404 returned error can't find the container with id 54419ec6348582247bf76d9d66987e86b48d93618de590b18c0d8630d9dbef2d Nov 01 22:30:10.176395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:10.176368 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" Nov 01 22:30:10.200150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:10.200113 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" event={"ID":"74dc6f3c-838b-4ea7-9b81-3f0890e8955b","Type":"ContainerStarted","Data":"6714cc2575d53764c5146bf62643fe5f294b8f96297915c2760633e36afe2f57"} Nov 01 22:30:10.201536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:10.201510 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" event={"ID":"2af45373-36c6-4d05-93c1-1a0a1805e7c2","Type":"ContainerStarted","Data":"54419ec6348582247bf76d9d66987e86b48d93618de590b18c0d8630d9dbef2d"} Nov 01 22:30:10.221348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:10.221295 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7l5cw" podStartSLOduration=1.9758556760000001 podStartE2EDuration="6.221280593s" totalImagesPullingTime="4.245424917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:10.220234848 +0000 UTC m=+375.648112775" watchObservedRunningTime="2026-11-01 22:30:10.221280593 +0000 UTC m=+375.649158523" Nov 01 22:30:10.631834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:10.631722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:11.431771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:11.431739 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:11.636382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:11.636348 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:11.968480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:11.968450 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:13.739683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:13.739647 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:13.841857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:13.841820 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:14.523029 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:30:14.522992 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:30:14.773469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:14.773433 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:15.225875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:15.225838 4981 generic.go:356] "Generic (PLEG): container finished" podID="2af45373-36c6-4d05-93c1-1a0a1805e7c2" containerID="1e0077586e2bc75803bfc12dffe829f7dc541a16c0bd00357675df95a22fb68c" exitCode=0 Nov 01 22:30:15.226145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:15.225902 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" event={"ID":"2af45373-36c6-4d05-93c1-1a0a1805e7c2","Type":"ContainerDied","Data":"1e0077586e2bc75803bfc12dffe829f7dc541a16c0bd00357675df95a22fb68c"} Nov 01 22:30:15.384537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:15.384501 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:16.231712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:16.231668 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" event={"ID":"2af45373-36c6-4d05-93c1-1a0a1805e7c2","Type":"ContainerStarted","Data":"897c3aee61fe9f798bcc028ae1351ac2d16c585f433eacc119fb7021c1c03292"} Nov 01 22:30:16.232195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:16.231806 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:16.289622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:16.289546 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" podStartSLOduration=1.673943365 podStartE2EDuration="7.289532016s" totalImagesPullingTime="5.157891689s" totalInitContainerRuntime="457.696962ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:16.285736744 +0000 UTC m=+381.713614676" watchObservedRunningTime="2026-11-01 22:30:16.289532016 +0000 UTC m=+381.717409947" Nov 01 22:30:19.234997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:19.234958 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:20.399460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:20.399431 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:20.498496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:20.498459 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:21.203870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:21.203836 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:25.411031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.410999 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf"] Nov 01 22:30:25.419402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.419381 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.423258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.423234 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf"] Nov 01 22:30:25.425794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.425714 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 22:30:25.425794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.425756 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 22:30:25.425999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.425829 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 22:30:25.425999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.425830 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 22:30:25.425999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.425887 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2m2x7\"" Nov 01 22:30:25.436322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.436298 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:25.486877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.486849 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b82fae95-f18b-4064-a70b-3cf1a89efc73-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.487053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.486885 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82fae95-f18b-4064-a70b-3cf1a89efc73-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.487053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.486908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4bm\" (UniqueName: \"kubernetes.io/projected/b82fae95-f18b-4064-a70b-3cf1a89efc73-kube-api-access-kq4bm\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.539297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.539255 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:25.588366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.588324 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b82fae95-f18b-4064-a70b-3cf1a89efc73-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.588366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.588369 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82fae95-f18b-4064-a70b-3cf1a89efc73-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.588597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.588395 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4bm\" (UniqueName: \"kubernetes.io/projected/b82fae95-f18b-4064-a70b-3cf1a89efc73-kube-api-access-kq4bm\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.589099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.589074 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b82fae95-f18b-4064-a70b-3cf1a89efc73-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.590914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.590889 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b82fae95-f18b-4064-a70b-3cf1a89efc73-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.596383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.596358 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4bm\" (UniqueName: \"kubernetes.io/projected/b82fae95-f18b-4064-a70b-3cf1a89efc73-kube-api-access-kq4bm\") pod \"kuadrant-console-plugin-66c4fb575d-mqgpf\" (UID: \"b82fae95-f18b-4064-a70b-3cf1a89efc73\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:25.638778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.638746 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:25.731698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:25.731662 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" Nov 01 22:30:26.000197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:26.000175 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf"] Nov 01 22:30:26.002631 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:30:26.002567 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb82fae95_f18b_4064_a70b_3cf1a89efc73.slice/crio-6cc94f7042bff3f4a67f25ac026aaa39c09235d6be1689253bc4797e2563089c WatchSource:0}: Error finding container 6cc94f7042bff3f4a67f25ac026aaa39c09235d6be1689253bc4797e2563089c: Status 404 returned error can't find the container with id 6cc94f7042bff3f4a67f25ac026aaa39c09235d6be1689253bc4797e2563089c Nov 01 22:30:26.280979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:26.280889 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" event={"ID":"b82fae95-f18b-4064-a70b-3cf1a89efc73","Type":"ContainerStarted","Data":"6cc94f7042bff3f4a67f25ac026aaa39c09235d6be1689253bc4797e2563089c"} Nov 01 22:30:26.386070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:26.386038 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.458284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:26.458256 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.779876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:26.779844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:26.853718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:26.853687 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:27.243981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:27.243927 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b59bbd84f-k42k8" Nov 01 22:30:28.235682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:28.235655 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:28.311249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:28.311219 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:32.649182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:32.649146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:32.838149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:32.838107 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:34.325020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:34.324981 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" event={"ID":"b82fae95-f18b-4064-a70b-3cf1a89efc73","Type":"ContainerStarted","Data":"764dd7839d1a4d4dadc04a263782b83c2beae3df8571fa6eccb848bfef35b66d"} Nov 01 22:30:34.346287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:34.346230 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mqgpf" podStartSLOduration=1.927576879 podStartE2EDuration="9.346217313s" totalImagesPullingTime="7.418640434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:34.344621834 +0000 UTC m=+399.772499765" watchObservedRunningTime="2026-11-01 22:30:34.346217313 +0000 UTC m=+399.774095243" Nov 01 22:30:39.616456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:39.616420 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:39.700077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:39.700043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:40.045042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.045008 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:40.158124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.158089 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:40.322634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.322534 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:40.369959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.369931 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq"] Nov 01 22:30:40.373535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.373514 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.384691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.384666 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-875b7\"" Nov 01 22:30:40.385311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.385300 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 22:30:40.385311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.385302 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 22:30:40.385395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.385317 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 22:30:40.385395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.385344 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 22:30:40.393792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.393766 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq"] Nov 01 22:30:40.523456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.523423 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-tmp\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.523671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.523463 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvnqw\" (UniqueName: \"kubernetes.io/projected/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-kube-api-access-kvnqw\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.523748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.523687 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-tls-certs\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.532738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.532711 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:40.624315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.624230 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-tls-certs\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.624315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.624275 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-tmp\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.624315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.624308 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvnqw\" (UniqueName: \"kubernetes.io/projected/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-kube-api-access-kvnqw\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.626667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.626640 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-tmp\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.626780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.626732 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-tls-certs\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.699856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.699827 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvnqw\" (UniqueName: \"kubernetes.io/projected/309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a-kube-api-access-kvnqw\") pod \"kube-auth-proxy-9bc8df847-pjzcq\" (UID: \"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a\") " pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.984901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.984869 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" Nov 01 22:30:40.986125 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:40.986096 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:41.200010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.199944 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:41.268174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.268058 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq"] Nov 01 22:30:41.271682 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:30:41.271650 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309eb3d8_3c5b_4ee7_bc7e_44b6ab6d7a8a.slice/crio-f1ef7801856e65503e50f327838c065493ad60261cfdcff1e6b869d3079c85ee WatchSource:0}: Error finding container f1ef7801856e65503e50f327838c065493ad60261cfdcff1e6b869d3079c85ee: Status 404 returned error can't find the container with id f1ef7801856e65503e50f327838c065493ad60261cfdcff1e6b869d3079c85ee Nov 01 22:30:41.355319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.355242 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" event={"ID":"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a","Type":"ContainerStarted","Data":"f1ef7801856e65503e50f327838c065493ad60261cfdcff1e6b869d3079c85ee"} Nov 01 22:30:41.394518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.394484 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:41.677054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.677012 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:41.879270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.879233 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z7p2b"] Nov 01 22:30:41.882821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.882797 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:41.895058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.895026 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 22:30:41.897276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.897252 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z7p2b"] Nov 01 22:30:41.936907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:41.936824 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:42.025208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.025177 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z7p2b"] Nov 01 22:30:42.035427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.035394 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvcw\" (UniqueName: \"kubernetes.io/projected/8ae05f33-3c13-4839-bed9-b437adee6799-kube-api-access-8gvcw\") pod \"limitador-limitador-74dfcfb894-z7p2b\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:42.035604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.035587 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ae05f33-3c13-4839-bed9-b437adee6799-config-file\") pod \"limitador-limitador-74dfcfb894-z7p2b\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:42.136008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.135971 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvcw\" (UniqueName: \"kubernetes.io/projected/8ae05f33-3c13-4839-bed9-b437adee6799-kube-api-access-8gvcw\") pod \"limitador-limitador-74dfcfb894-z7p2b\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:42.136008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.136023 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ae05f33-3c13-4839-bed9-b437adee6799-config-file\") pod \"limitador-limitador-74dfcfb894-z7p2b\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:42.137258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.137231 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ae05f33-3c13-4839-bed9-b437adee6799-config-file\") pod \"limitador-limitador-74dfcfb894-z7p2b\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:42.153068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.153039 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvcw\" (UniqueName: \"kubernetes.io/projected/8ae05f33-3c13-4839-bed9-b437adee6799-kube-api-access-8gvcw\") pod \"limitador-limitador-74dfcfb894-z7p2b\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:42.200596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.200512 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:42.219949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.219924 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:42.520091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.520015 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z7p2b"] Nov 01 22:30:42.524495 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:30:42.524468 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae05f33_3c13_4839_bed9_b437adee6799.slice/crio-6a43ea93dba482b17fa3cb7c2b0eda6518b78853517f97f23de65ef7919021b3 WatchSource:0}: Error finding container 6a43ea93dba482b17fa3cb7c2b0eda6518b78853517f97f23de65ef7919021b3: Status 404 returned error can't find the container with id 6a43ea93dba482b17fa3cb7c2b0eda6518b78853517f97f23de65ef7919021b3 Nov 01 22:30:42.623759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.623722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:42.867797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.867717 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj"] Nov 01 22:30:42.876802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.876774 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:42.889894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.889859 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 01 22:30:42.890064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.889932 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 01 22:30:42.890064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.889979 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 01 22:30:42.890376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.890352 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj"] Nov 01 22:30:42.890921 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.890829 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-qnzjg\"" Nov 01 22:30:42.942677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.942642 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x9r\" (UniqueName: \"kubernetes.io/projected/b3464466-11fd-4690-bd6a-7d6a80854fac-kube-api-access-z8x9r\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:42.942867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.942726 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3464466-11fd-4690-bd6a-7d6a80854fac-cert\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:42.942867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.942821 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3464466-11fd-4690-bd6a-7d6a80854fac-manager-config\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:42.942867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:42.942855 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3464466-11fd-4690-bd6a-7d6a80854fac-metrics-certs\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.043440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.043406 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3464466-11fd-4690-bd6a-7d6a80854fac-manager-config\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.043440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.043448 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3464466-11fd-4690-bd6a-7d6a80854fac-metrics-certs\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.043718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.043517 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x9r\" (UniqueName: \"kubernetes.io/projected/b3464466-11fd-4690-bd6a-7d6a80854fac-kube-api-access-z8x9r\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.043718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.043571 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3464466-11fd-4690-bd6a-7d6a80854fac-cert\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.044141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.044116 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3464466-11fd-4690-bd6a-7d6a80854fac-manager-config\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.046630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.046603 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3464466-11fd-4690-bd6a-7d6a80854fac-metrics-certs\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.046901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.046877 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3464466-11fd-4690-bd6a-7d6a80854fac-cert\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.058397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.058369 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x9r\" (UniqueName: \"kubernetes.io/projected/b3464466-11fd-4690-bd6a-7d6a80854fac-kube-api-access-z8x9r\") pod \"jobset-controller-manager-656887d9d4-psjhj\" (UID: \"b3464466-11fd-4690-bd6a-7d6a80854fac\") " pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.112878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.112830 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:43.203016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.202979 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:43.244080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.244043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:43.365506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.365468 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" event={"ID":"8ae05f33-3c13-4839-bed9-b437adee6799","Type":"ContainerStarted","Data":"6a43ea93dba482b17fa3cb7c2b0eda6518b78853517f97f23de65ef7919021b3"} Nov 01 22:30:43.524362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:43.524284 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj"] Nov 01 22:30:43.528222 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:30:43.528194 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3464466_11fd_4690_bd6a_7d6a80854fac.slice/crio-57b5e20757f5925b31a57ca7156d5c934589312584c7b62c4f3fffd36bf676a3 WatchSource:0}: Error finding container 57b5e20757f5925b31a57ca7156d5c934589312584c7b62c4f3fffd36bf676a3: Status 404 returned error can't find the container with id 57b5e20757f5925b31a57ca7156d5c934589312584c7b62c4f3fffd36bf676a3 Nov 01 22:30:44.372195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:44.372153 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" event={"ID":"b3464466-11fd-4690-bd6a-7d6a80854fac","Type":"ContainerStarted","Data":"57b5e20757f5925b31a57ca7156d5c934589312584c7b62c4f3fffd36bf676a3"} Nov 01 22:30:44.526212 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:30:44.526180 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:30:45.273763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:45.273735 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:45.377610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:45.377561 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:45.378488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:45.378454 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" event={"ID":"309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a","Type":"ContainerStarted","Data":"640fc03c07f6a08784a05dc41329a3c3636739e4565f799bd0de416ac98ba1b5"} Nov 01 22:30:45.397643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:45.397565 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9bc8df847-pjzcq" podStartSLOduration=2.094753729 podStartE2EDuration="5.397548684s" totalImagesPullingTime="3.302794955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:45.395633367 +0000 UTC m=+410.823511299" watchObservedRunningTime="2026-11-01 22:30:45.397548684 +0000 UTC m=+410.825426616" Nov 01 22:30:46.385270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:46.385233 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" event={"ID":"8ae05f33-3c13-4839-bed9-b437adee6799","Type":"ContainerStarted","Data":"eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8"} Nov 01 22:30:46.430097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:46.430039 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" podStartSLOduration=1.769909476 podStartE2EDuration="5.430025529s" totalImagesPullingTime="3.660116053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:46.428734066 +0000 UTC m=+411.856611997" watchObservedRunningTime="2026-11-01 22:30:46.430025529 +0000 UTC m=+411.857903438" Nov 01 22:30:47.390534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:47.390496 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:50.441667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:50.441629 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:50.525809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:50.525764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:51.020753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:51.020715 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:51.121211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:51.121178 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:51.626436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:51.626401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:51.726251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:51.726223 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:52.412840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:52.412798 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" event={"ID":"b3464466-11fd-4690-bd6a-7d6a80854fac","Type":"ContainerStarted","Data":"62439c16f9ff534e6e8d67836030fdf91b992239d003edf6da1b41c8809e1e95"} Nov 01 22:30:52.413027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:52.412938 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:30:52.433055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:52.433011 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" podStartSLOduration=2.559814611 podStartE2EDuration="10.432997663s" totalImagesPullingTime="7.873183052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:52.431290624 +0000 UTC m=+417.859168557" watchObservedRunningTime="2026-11-01 22:30:52.432997663 +0000 UTC m=+417.860875604" Nov 01 22:30:58.211921 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:58.211883 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:58.288246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:58.288204 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:30:58.396557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:58.396526 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:59.434830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:59.434791 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z7p2b"] Nov 01 22:30:59.435245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:59.434999 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" podUID="8ae05f33-3c13-4839-bed9-b437adee6799" containerName="limitador" containerID="cri-o://eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8" gracePeriod=30 Nov 01 22:30:59.970646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:59.970619 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:30:59.988680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:59.988636 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae05f33-3c13-4839-bed9-b437adee6799-config-file\" (UniqueName: \"kubernetes.io/configmap/8ae05f33-3c13-4839-bed9-b437adee6799-config-file\") pod \"8ae05f33-3c13-4839-bed9-b437adee6799\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " Nov 01 22:30:59.988817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:59.988717 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae05f33-3c13-4839-bed9-b437adee6799-kube-api-access-8gvcw\" (UniqueName: \"kubernetes.io/projected/8ae05f33-3c13-4839-bed9-b437adee6799-kube-api-access-8gvcw\") pod \"8ae05f33-3c13-4839-bed9-b437adee6799\" (UID: \"8ae05f33-3c13-4839-bed9-b437adee6799\") " Nov 01 22:30:59.989065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:59.989034 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae05f33-3c13-4839-bed9-b437adee6799-config-file" pod "8ae05f33-3c13-4839-bed9-b437adee6799" (UID: "8ae05f33-3c13-4839-bed9-b437adee6799"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:30:59.991299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:30:59.991277 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae05f33-3c13-4839-bed9-b437adee6799-kube-api-access-8gvcw" pod "8ae05f33-3c13-4839-bed9-b437adee6799" (UID: "8ae05f33-3c13-4839-bed9-b437adee6799"). InnerVolumeSpecName "kube-api-access-8gvcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:31:00.089899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.089861 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gvcw\" (UniqueName: \"kubernetes.io/projected/8ae05f33-3c13-4839-bed9-b437adee6799-kube-api-access-8gvcw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.089899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.089891 4981 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8ae05f33-3c13-4839-bed9-b437adee6799-config-file\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:31:00.447161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.447132 4981 generic.go:356] "Generic (PLEG): container finished" podID="8ae05f33-3c13-4839-bed9-b437adee6799" containerID="eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8" exitCode=0 Nov 01 22:31:00.447608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.447184 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" Nov 01 22:31:00.447608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.447193 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" event={"ID":"8ae05f33-3c13-4839-bed9-b437adee6799","Type":"ContainerDied","Data":"eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8"} Nov 01 22:31:00.447608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.447237 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-z7p2b" event={"ID":"8ae05f33-3c13-4839-bed9-b437adee6799","Type":"ContainerDied","Data":"6a43ea93dba482b17fa3cb7c2b0eda6518b78853517f97f23de65ef7919021b3"} Nov 01 22:31:00.447608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.447258 4981 scope.go:122] "RemoveContainer" containerID="eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8" Nov 01 22:31:00.455614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.455592 4981 scope.go:122] "RemoveContainer" containerID="eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8" Nov 01 22:31:00.455866 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:31:00.455848 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8\": container with ID starting with eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8 not found: ID does not exist" containerID="eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8" Nov 01 22:31:00.455926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.455877 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8"} err="failed to get container status \"eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8\": rpc error: code = NotFound desc = could not find container \"eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8\": container with ID starting with eaa99938f65a4eed66eda3864c775dcf703f4643d4154ce031ea7507ee6307e8 not found: ID does not exist" Nov 01 22:31:00.468341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.468316 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z7p2b"] Nov 01 22:31:00.471621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.471599 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-z7p2b"] Nov 01 22:31:00.732018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:00.731942 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae05f33-3c13-4839-bed9-b437adee6799" path="/var/lib/kubelet/pods/8ae05f33-3c13-4839-bed9-b437adee6799/volumes" Nov 01 22:31:03.422108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:03.422077 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" Nov 01 22:31:04.031992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:04.031953 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:04.132015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:04.131981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:06.631705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:06.631673 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:06.732763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:06.732732 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:07.757850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:07.757818 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:07.858197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:07.858168 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:08.533080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:08.533047 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:10.831020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:10.830983 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:10.931108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:10.931079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:11.031263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:11.031220 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:11.104621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:11.104523 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:12.019474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:12.019434 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:12.131321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:12.131289 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:14.522490 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:31:14.522460 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:31:15.558799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:15.558768 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:15.660017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:15.659990 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:17.488127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:17.488097 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:17.614743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:17.614710 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:23.968934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:23.968895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:24.049055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:24.049019 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:28.095204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:28.095165 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:28.183496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:28.183463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:28.262052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:28.262020 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:28.363002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:28.362916 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:32.104946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:32.104914 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:32.254156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:32.254117 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.360420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:33.360386 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:33.489470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:33.489440 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:39.410706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.410630 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:39.587001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.586959 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:39.660470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.657917 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cxr2r"] Nov 01 22:31:39.660470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.658721 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8ae05f33-3c13-4839-bed9-b437adee6799" containerName="limitador" Nov 01 22:31:39.660470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.658740 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae05f33-3c13-4839-bed9-b437adee6799" containerName="limitador" Nov 01 22:31:39.660470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.658892 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae05f33-3c13-4839-bed9-b437adee6799" containerName="limitador" Nov 01 22:31:39.662596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.662524 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:39.684757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.684737 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 22:31:39.694885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.694866 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 22:31:39.696526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.696501 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rkv2n\"" Nov 01 22:31:39.696667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.696522 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 22:31:39.698146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.698124 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cxr2r"] Nov 01 22:31:39.717493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.717470 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-8wbsw"] Nov 01 22:31:39.720859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.720844 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.753841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.753815 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8wbsw"] Nov 01 22:31:39.800458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.800431 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 22:31:39.800458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.800449 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 22:31:39.800664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.800449 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wrwcn\"" Nov 01 22:31:39.840918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.840882 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7jt\" (UniqueName: \"kubernetes.io/projected/80467d4e-cd6c-479b-8334-daeb938706b1-kube-api-access-dv7jt\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.841082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.840949 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80467d4e-cd6c-479b-8334-daeb938706b1-metrics-certs\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.841082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.841057 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80467d4e-cd6c-479b-8334-daeb938706b1-cert\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.841258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.841114 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcd9\" (UniqueName: \"kubernetes.io/projected/1406f61b-2ddc-402f-8631-3b00159964b0-kube-api-access-dlcd9\") pod \"model-serving-api-75bd89fdfc-cxr2r\" (UID: \"1406f61b-2ddc-402f-8631-3b00159964b0\") " pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:39.841258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.841220 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1406f61b-2ddc-402f-8631-3b00159964b0-tls-certs\") pod \"model-serving-api-75bd89fdfc-cxr2r\" (UID: \"1406f61b-2ddc-402f-8631-3b00159964b0\") " pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:39.885470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.885417 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:39.942251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.942223 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80467d4e-cd6c-479b-8334-daeb938706b1-metrics-certs\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.942393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.942263 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80467d4e-cd6c-479b-8334-daeb938706b1-cert\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.942393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.942307 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcd9\" (UniqueName: \"kubernetes.io/projected/1406f61b-2ddc-402f-8631-3b00159964b0-kube-api-access-dlcd9\") pod \"model-serving-api-75bd89fdfc-cxr2r\" (UID: \"1406f61b-2ddc-402f-8631-3b00159964b0\") " pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:39.942393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.942336 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1406f61b-2ddc-402f-8631-3b00159964b0-tls-certs\") pod \"model-serving-api-75bd89fdfc-cxr2r\" (UID: \"1406f61b-2ddc-402f-8631-3b00159964b0\") " pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:39.942393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.942366 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7jt\" (UniqueName: \"kubernetes.io/projected/80467d4e-cd6c-479b-8334-daeb938706b1-kube-api-access-dv7jt\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.944881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.944857 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1406f61b-2ddc-402f-8631-3b00159964b0-tls-certs\") pod \"model-serving-api-75bd89fdfc-cxr2r\" (UID: \"1406f61b-2ddc-402f-8631-3b00159964b0\") " pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:39.944999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.944887 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80467d4e-cd6c-479b-8334-daeb938706b1-cert\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.944999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.944904 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80467d4e-cd6c-479b-8334-daeb938706b1-metrics-certs\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.953283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.953262 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcd9\" (UniqueName: \"kubernetes.io/projected/1406f61b-2ddc-402f-8631-3b00159964b0-kube-api-access-dlcd9\") pod \"model-serving-api-75bd89fdfc-cxr2r\" (UID: \"1406f61b-2ddc-402f-8631-3b00159964b0\") " pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:39.972950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.972925 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7jt\" (UniqueName: \"kubernetes.io/projected/80467d4e-cd6c-479b-8334-daeb938706b1-kube-api-access-dv7jt\") pod \"odh-model-controller-769875db56-8wbsw\" (UID: \"80467d4e-cd6c-479b-8334-daeb938706b1\") " pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:39.975525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:39.975509 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:40.033922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:40.033890 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:40.190596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:40.190547 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:40.258649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:40.258624 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cxr2r"] Nov 01 22:31:40.261046 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:31:40.261012 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1406f61b_2ddc_402f_8631_3b00159964b0.slice/crio-aac1d3c2a5eed829d0b5c8cc317bb1cd8b2620fa8669e3d1fc7d364c86c4a264 WatchSource:0}: Error finding container aac1d3c2a5eed829d0b5c8cc317bb1cd8b2620fa8669e3d1fc7d364c86c4a264: Status 404 returned error can't find the container with id aac1d3c2a5eed829d0b5c8cc317bb1cd8b2620fa8669e3d1fc7d364c86c4a264 Nov 01 22:31:40.317298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:40.317273 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-8wbsw"] Nov 01 22:31:40.318954 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:31:40.318926 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80467d4e_cd6c_479b_8334_daeb938706b1.slice/crio-58fe964f384b4e4b852b3b419bd3f2db8f2309bb1b1d56fc3964b5643c72c341 WatchSource:0}: Error finding container 58fe964f384b4e4b852b3b419bd3f2db8f2309bb1b1d56fc3964b5643c72c341: Status 404 returned error can't find the container with id 58fe964f384b4e4b852b3b419bd3f2db8f2309bb1b1d56fc3964b5643c72c341 Nov 01 22:31:40.613832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:40.613741 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" event={"ID":"1406f61b-2ddc-402f-8631-3b00159964b0","Type":"ContainerStarted","Data":"aac1d3c2a5eed829d0b5c8cc317bb1cd8b2620fa8669e3d1fc7d364c86c4a264"} Nov 01 22:31:40.614897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:40.614876 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8wbsw" event={"ID":"80467d4e-cd6c-479b-8334-daeb938706b1","Type":"ContainerStarted","Data":"58fe964f384b4e4b852b3b419bd3f2db8f2309bb1b1d56fc3964b5643c72c341"} Nov 01 22:31:41.132830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:41.132788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:41.246193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:41.246102 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:41.486984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:41.486953 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:31:42.627457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:42.627363 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" event={"ID":"1406f61b-2ddc-402f-8631-3b00159964b0","Type":"ContainerStarted","Data":"0c4aee3abf11b9bb9291d1cca04e1e88485cab3e76b150d036ee64be5ebf232c"} Nov 01 22:31:42.627457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:42.627440 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:42.652650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:42.652601 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" podStartSLOduration=1.561807606 podStartE2EDuration="3.652585563s" totalImagesPullingTime="2.090777957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:42.650545905 +0000 UTC m=+468.078423838" watchObservedRunningTime="2026-11-01 22:31:42.652585563 +0000 UTC m=+468.080463485" Nov 01 22:31:43.632703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:43.632665 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-8wbsw" event={"ID":"80467d4e-cd6c-479b-8334-daeb938706b1","Type":"ContainerStarted","Data":"365ffc859afc07c965e56a52212bdc79f9ba5e6ec6e5299a5e25110397e4f0d5"} Nov 01 22:31:43.633269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:43.632820 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:31:43.653968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:43.653904 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-8wbsw" podStartSLOduration=1.486431751 podStartE2EDuration="4.653884375s" totalImagesPullingTime="3.167452624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:43.65267585 +0000 UTC m=+469.080553781" watchObservedRunningTime="2026-11-01 22:31:43.653884375 +0000 UTC m=+469.081762311" Nov 01 22:31:44.522942 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:31:44.522902 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:31:53.636614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:53.636561 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" Nov 01 22:31:54.641448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:31:54.641418 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-8wbsw" Nov 01 22:32:03.453109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:03.453067 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:03.562448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:03.562413 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:04.762817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:04.762790 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:04.882461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:04.882429 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:04.994866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:04.994829 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:12.103724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:12.103691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:14.522590 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:32:14.522541 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:32:15.132065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:15.132009 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:15.233347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:15.233299 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:17.574360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:17.574314 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:17.675787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:17.675753 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:40.803159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:40.803121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:40.892543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:40.892507 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:44.523861 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:32:44.523758 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:32:52.620211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:52.620169 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:32:52.726266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:32:52.726231 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:11.866460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:11.866363 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:11.968640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:11.968598 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:12.112261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:12.112227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:14.523473 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:33:14.523434 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:33:30.185681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:30.185648 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:31.300452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.300418 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m"] Nov 01 22:33:31.304164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.304139 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.313149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.313131 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 22:33:31.313291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.313273 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 22:33:31.314623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.314606 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-njcj2\"" Nov 01 22:33:31.314688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.314644 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 01 22:33:31.318677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.318652 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m"] Nov 01 22:33:31.347520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.347486 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:31.389716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.389679 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz2vf\" (UniqueName: \"kubernetes.io/projected/c6f43edf-54c6-41d3-8445-a8d2aa81a230-kube-api-access-nz2vf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.389885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.389719 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.389885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.389805 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.389885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.389860 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.389997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.389928 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f43edf-54c6-41d3-8445-a8d2aa81a230-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.389997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.389982 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.490978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.490941 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz2vf\" (UniqueName: \"kubernetes.io/projected/c6f43edf-54c6-41d3-8445-a8d2aa81a230-kube-api-access-nz2vf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.490978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.490976 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.491204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.491005 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.491204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.491033 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.491204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.491057 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f43edf-54c6-41d3-8445-a8d2aa81a230-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.491204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.491086 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.491473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.491446 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.491473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.491462 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.491596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.491549 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.493442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.493423 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f43edf-54c6-41d3-8445-a8d2aa81a230-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.493764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.493746 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f43edf-54c6-41d3-8445-a8d2aa81a230-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.499135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.499111 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz2vf\" (UniqueName: \"kubernetes.io/projected/c6f43edf-54c6-41d3-8445-a8d2aa81a230-kube-api-access-nz2vf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m\" (UID: \"c6f43edf-54c6-41d3-8445-a8d2aa81a230\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.617524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.617427 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:31.889866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:31.889806 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m"] Nov 01 22:33:31.892584 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:33:31.892542 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f43edf_54c6_41d3_8445_a8d2aa81a230.slice/crio-0d4119a809574a3166452a351f08f0b0e571390ea7e27f52bd7d1aab10323e7c WatchSource:0}: Error finding container 0d4119a809574a3166452a351f08f0b0e571390ea7e27f52bd7d1aab10323e7c: Status 404 returned error can't find the container with id 0d4119a809574a3166452a351f08f0b0e571390ea7e27f52bd7d1aab10323e7c Nov 01 22:33:32.078853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:32.078816 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" event={"ID":"c6f43edf-54c6-41d3-8445-a8d2aa81a230","Type":"ContainerStarted","Data":"0d4119a809574a3166452a351f08f0b0e571390ea7e27f52bd7d1aab10323e7c"} Nov 01 22:33:32.582193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:32.582154 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:33.666668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:33.666633 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:34.098619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.098563 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" event={"ID":"c6f43edf-54c6-41d3-8445-a8d2aa81a230","Type":"ContainerStarted","Data":"128366f8f1ab57dbe287b8928875fc598c81e1d725c092e1ff4b2434971031bc"} Nov 01 22:33:34.116383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.116327 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" podStartSLOduration=1.9808963689999999 podStartE2EDuration="3.116313603s" totalImagesPullingTime="1.135417234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:34.115390497 +0000 UTC m=+579.543268428" watchObservedRunningTime="2026-11-01 22:33:34.116313603 +0000 UTC m=+579.544191533" Nov 01 22:33:34.579976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.579942 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8"] Nov 01 22:33:34.583717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.583693 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.592771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.592744 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 22:33:34.595081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.595053 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8"] Nov 01 22:33:34.618548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.618504 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:34.647283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.647250 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:34.729028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.728992 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.729436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.729038 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.729436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.729147 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cmk\" (UniqueName: \"kubernetes.io/projected/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-kube-api-access-j9cmk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.729436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.729172 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.729436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.729288 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.729436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.729314 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.829837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.829800 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.829837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.829840 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.830106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.829886 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.830106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.829903 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.830106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.829929 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cmk\" (UniqueName: \"kubernetes.io/projected/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-kube-api-access-j9cmk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.830106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.829950 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.830342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.830243 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.830402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.830381 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.830448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.830412 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.832202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.832182 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.832372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.832354 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.842121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.842096 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cmk\" (UniqueName: \"kubernetes.io/projected/e7f7e966-fddb-4b4c-b76e-f2c2f55c8350-kube-api-access-j9cmk\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8\" (UID: \"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:34.897054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:34.897019 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:35.169540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:35.169333 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8"] Nov 01 22:33:35.172183 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:33:35.172158 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7f7e966_fddb_4b4c_b76e_f2c2f55c8350.slice/crio-d4fd37a1af14182d805e6156f4e5806ec9b15d4d4cf1e3669ec663898d7a340b WatchSource:0}: Error finding container d4fd37a1af14182d805e6156f4e5806ec9b15d4d4cf1e3669ec663898d7a340b: Status 404 returned error can't find the container with id d4fd37a1af14182d805e6156f4e5806ec9b15d4d4cf1e3669ec663898d7a340b Nov 01 22:33:36.030062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.030028 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn"] Nov 01 22:33:36.033493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.033470 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.039931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.039906 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 01 22:33:36.047641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.047562 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn"] Nov 01 22:33:36.090250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.090217 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:36.108883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.108847 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" event={"ID":"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350","Type":"ContainerStarted","Data":"8d3087a8d816db742e8082d1b43772536936595f49f3ae372a4d62742b82920a"} Nov 01 22:33:36.108883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.108889 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" event={"ID":"e7f7e966-fddb-4b4c-b76e-f2c2f55c8350","Type":"ContainerStarted","Data":"d4fd37a1af14182d805e6156f4e5806ec9b15d4d4cf1e3669ec663898d7a340b"} Nov 01 22:33:36.128082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.128007 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podStartSLOduration=1.856449027 podStartE2EDuration="2.127995458s" totalImagesPullingTime="271.546431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:36.127060162 +0000 UTC m=+581.554938096" watchObservedRunningTime="2026-11-01 22:33:36.127995458 +0000 UTC m=+581.555873388" Nov 01 22:33:36.141555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.141523 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltq6d\" (UniqueName: \"kubernetes.io/projected/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-kube-api-access-ltq6d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.141723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.141600 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.141723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.141681 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.141794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.141737 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.141827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.141798 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.141827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.141824 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.242973 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243033 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243062 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243077 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243161 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltq6d\" (UniqueName: \"kubernetes.io/projected/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-kube-api-access-ltq6d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243273 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243451 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243471 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.243727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.243702 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.245530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.245506 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.245798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.245780 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.252561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.252536 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltq6d\" (UniqueName: \"kubernetes.io/projected/7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23-kube-api-access-ltq6d\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn\" (UID: \"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.347342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.347260 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:36.529938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.529902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:36.624225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.624197 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn"] Nov 01 22:33:36.625981 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:33:36.625949 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7109922b_7a0a_4ba7_b7a6_a49dcf5e6a23.slice/crio-330c66e45012b433fb3b437ba07100aa0e0740942a0e1aef79b19f1a3c404748 WatchSource:0}: Error finding container 330c66e45012b433fb3b437ba07100aa0e0740942a0e1aef79b19f1a3c404748: Status 404 returned error can't find the container with id 330c66e45012b433fb3b437ba07100aa0e0740942a0e1aef79b19f1a3c404748 Nov 01 22:33:36.897715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:36.897640 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:37.114677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:37.114637 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" event={"ID":"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23","Type":"ContainerStarted","Data":"b4ed8b8c169d62ece740e1a6fdd8cd4d016c421cde66dd1bcb0b78eee3666632"} Nov 01 22:33:37.115144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:37.114715 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" event={"ID":"7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23","Type":"ContainerStarted","Data":"330c66e45012b433fb3b437ba07100aa0e0740942a0e1aef79b19f1a3c404748"} Nov 01 22:33:37.133554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:37.133505 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" podStartSLOduration=0.843955143 podStartE2EDuration="1.133490977s" totalImagesPullingTime="289.535834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:37.132640016 +0000 UTC m=+582.560517947" watchObservedRunningTime="2026-11-01 22:33:37.133490977 +0000 UTC m=+582.561368907" Nov 01 22:33:37.347662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:37.347623 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:39.304146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:39.304112 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:41.617827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:41.617790 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:41.630425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:41.630398 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:42.149309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:42.149277 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:33:44.522389 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:33:44.522352 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:33:44.898198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:44.898111 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:44.910617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:44.910568 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:45.163337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:45.163264 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:33:46.348053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:46.348025 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:46.360533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:46.360505 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:47.173181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:47.173154 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" Nov 01 22:33:52.881146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.881114 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-h8248"] Nov 01 22:33:52.887231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.887213 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:52.893296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.893267 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 22:33:52.893410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.893294 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 22:33:52.893410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.893345 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xpmzp\"" Nov 01 22:33:52.898191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.898168 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-h8248"] Nov 01 22:33:52.936409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.936381 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:52.968733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.968707 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7374fdc-9e86-410d-a9aa-3579563600b8-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-h8248\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:52.968830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:52.968753 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tnbf\" (UniqueName: \"kubernetes.io/projected/b7374fdc-9e86-410d-a9aa-3579563600b8-kube-api-access-7tnbf\") pod \"ai-gateway-controller-5f6dd4f69f-h8248\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:53.036738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:53.036710 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:53.069884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:53.069857 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7374fdc-9e86-410d-a9aa-3579563600b8-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-h8248\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:53.070011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:53.069901 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tnbf\" (UniqueName: \"kubernetes.io/projected/b7374fdc-9e86-410d-a9aa-3579563600b8-kube-api-access-7tnbf\") pod \"ai-gateway-controller-5f6dd4f69f-h8248\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:53.072253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:53.072230 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7374fdc-9e86-410d-a9aa-3579563600b8-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-h8248\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:53.078318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:53.078293 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tnbf\" (UniqueName: \"kubernetes.io/projected/b7374fdc-9e86-410d-a9aa-3579563600b8-kube-api-access-7tnbf\") pod \"ai-gateway-controller-5f6dd4f69f-h8248\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:53.209785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:53.209327 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:53.509335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:53.509267 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-h8248"] Nov 01 22:33:53.512003 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:33:53.511976 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7374fdc_9e86_410d_a9aa_3579563600b8.slice/crio-e6a2415b7fc0562a187abaabc1f98da2002d316e3468027c6eee26369b31556b WatchSource:0}: Error finding container e6a2415b7fc0562a187abaabc1f98da2002d316e3468027c6eee26369b31556b: Status 404 returned error can't find the container with id e6a2415b7fc0562a187abaabc1f98da2002d316e3468027c6eee26369b31556b Nov 01 22:33:54.196384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:54.196347 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" event={"ID":"b7374fdc-9e86-410d-a9aa-3579563600b8","Type":"ContainerStarted","Data":"e6a2415b7fc0562a187abaabc1f98da2002d316e3468027c6eee26369b31556b"} Nov 01 22:33:57.754130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:57.754106 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 22:33:58.213786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:58.213749 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" event={"ID":"b7374fdc-9e86-410d-a9aa-3579563600b8","Type":"ContainerStarted","Data":"8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08"} Nov 01 22:33:58.213958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:58.213825 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:33:58.233046 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:58.233003 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" podStartSLOduration=1.995437619 podStartE2EDuration="6.232990199s" totalImagesPullingTime="4.23755258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:58.231476281 +0000 UTC m=+603.659354211" watchObservedRunningTime="2026-11-01 22:33:58.232990199 +0000 UTC m=+603.660868156" Nov 01 22:33:59.421337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.421310 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:59.450064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.450026 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gzxqz"] Nov 01 22:33:59.453635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.453615 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.464798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.464774 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4hqrk\"" Nov 01 22:33:59.464911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.464824 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 22:33:59.465753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.465728 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gzxqz"] Nov 01 22:33:59.495384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.495358 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sph7r"] Nov 01 22:33:59.498733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.498711 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:33:59.512155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.512128 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sph7r"] Nov 01 22:33:59.517359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.517336 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tps27\" (UniqueName: \"kubernetes.io/projected/fd1ade81-64b5-4dcf-a5da-55632908568b-kube-api-access-tps27\") pod \"payload-processing-55fbbbf9dc-gzxqz\" (UID: \"fd1ade81-64b5-4dcf-a5da-55632908568b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.517454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.517365 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1ade81-64b5-4dcf-a5da-55632908568b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gzxqz\" (UID: \"fd1ade81-64b5-4dcf-a5da-55632908568b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.517504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.517464 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvpx\" (UniqueName: \"kubernetes.io/projected/d3deba2b-ac37-495d-bee9-e015774a9626-kube-api-access-9dvpx\") pod \"payload-pre-processing-7b98bb678c-sph7r\" (UID: \"d3deba2b-ac37-495d-bee9-e015774a9626\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:33:59.517504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.517491 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3deba2b-ac37-495d-bee9-e015774a9626-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sph7r\" (UID: \"d3deba2b-ac37-495d-bee9-e015774a9626\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:33:59.548050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.548018 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:33:59.617936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.617902 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvpx\" (UniqueName: \"kubernetes.io/projected/d3deba2b-ac37-495d-bee9-e015774a9626-kube-api-access-9dvpx\") pod \"payload-pre-processing-7b98bb678c-sph7r\" (UID: \"d3deba2b-ac37-495d-bee9-e015774a9626\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:33:59.617936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.617938 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3deba2b-ac37-495d-bee9-e015774a9626-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sph7r\" (UID: \"d3deba2b-ac37-495d-bee9-e015774a9626\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:33:59.618190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.617996 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tps27\" (UniqueName: \"kubernetes.io/projected/fd1ade81-64b5-4dcf-a5da-55632908568b-kube-api-access-tps27\") pod \"payload-processing-55fbbbf9dc-gzxqz\" (UID: \"fd1ade81-64b5-4dcf-a5da-55632908568b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.618190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.618014 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1ade81-64b5-4dcf-a5da-55632908568b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gzxqz\" (UID: \"fd1ade81-64b5-4dcf-a5da-55632908568b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.618771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.618751 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1ade81-64b5-4dcf-a5da-55632908568b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-gzxqz\" (UID: \"fd1ade81-64b5-4dcf-a5da-55632908568b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.618820 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.618783 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3deba2b-ac37-495d-bee9-e015774a9626-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-sph7r\" (UID: \"d3deba2b-ac37-495d-bee9-e015774a9626\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:33:59.630109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.630080 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvpx\" (UniqueName: \"kubernetes.io/projected/d3deba2b-ac37-495d-bee9-e015774a9626-kube-api-access-9dvpx\") pod \"payload-pre-processing-7b98bb678c-sph7r\" (UID: \"d3deba2b-ac37-495d-bee9-e015774a9626\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:33:59.630222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.630158 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tps27\" (UniqueName: \"kubernetes.io/projected/fd1ade81-64b5-4dcf-a5da-55632908568b-kube-api-access-tps27\") pod \"payload-processing-55fbbbf9dc-gzxqz\" (UID: \"fd1ade81-64b5-4dcf-a5da-55632908568b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.768585 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.768539 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:33:59.811825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:33:59.811792 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:34:00.047126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:00.047044 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-gzxqz"] Nov 01 22:34:00.050489 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:00.050459 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1ade81_64b5_4dcf_a5da_55632908568b.slice/crio-520c6d46babac91adae028b37f79551b306690ad18ca8b43ee6dd6b27f8b7afc WatchSource:0}: Error finding container 520c6d46babac91adae028b37f79551b306690ad18ca8b43ee6dd6b27f8b7afc: Status 404 returned error can't find the container with id 520c6d46babac91adae028b37f79551b306690ad18ca8b43ee6dd6b27f8b7afc Nov 01 22:34:00.097039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:00.097015 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-sph7r"] Nov 01 22:34:00.099847 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:00.099819 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3deba2b_ac37_495d_bee9_e015774a9626.slice/crio-f39392b623815dafbe68db04d1ba77d9954f8b46f607c435ce40a5c7bfbda356 WatchSource:0}: Error finding container f39392b623815dafbe68db04d1ba77d9954f8b46f607c435ce40a5c7bfbda356: Status 404 returned error can't find the container with id f39392b623815dafbe68db04d1ba77d9954f8b46f607c435ce40a5c7bfbda356 Nov 01 22:34:00.223668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:00.223611 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" event={"ID":"d3deba2b-ac37-495d-bee9-e015774a9626","Type":"ContainerStarted","Data":"f39392b623815dafbe68db04d1ba77d9954f8b46f607c435ce40a5c7bfbda356"} Nov 01 22:34:00.224853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:00.224826 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" event={"ID":"fd1ade81-64b5-4dcf-a5da-55632908568b","Type":"ContainerStarted","Data":"520c6d46babac91adae028b37f79551b306690ad18ca8b43ee6dd6b27f8b7afc"} Nov 01 22:34:01.005916 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.005877 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h"] Nov 01 22:34:01.010084 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.010061 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.020635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.020614 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 22:34:01.020738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.020663 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27s9n\"" Nov 01 22:34:01.024832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.024811 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h"] Nov 01 22:34:01.026530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.026508 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg"] Nov 01 22:34:01.030809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.030792 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.033304 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033284 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033357 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033401 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033461 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033486 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033547 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033605 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033635 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.033709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.033663 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jfwc\" (UniqueName: \"kubernetes.io/projected/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-kube-api-access-2jfwc\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.054195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.054155 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg"] Nov 01 22:34:01.134486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134447 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.134694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134604 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jfwc\" (UniqueName: \"kubernetes.io/projected/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-kube-api-access-2jfwc\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.134694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134656 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.134694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134685 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.134863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134716 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.134863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134749 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0369ac9e-dda1-4f8a-878a-24bddecac57b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.134863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134784 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5km\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-kube-api-access-nc5km\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.134863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134817 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.134863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134853 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134876 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134907 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.135162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.134913 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135222 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135240 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135363 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.135438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135424 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135483 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.135540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135517 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.135678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135608 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135639 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135668 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.135830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135745 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.135993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.135976 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.138366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.138342 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.138485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.138377 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.152170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.152148 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.160117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.160097 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jfwc\" (UniqueName: \"kubernetes.io/projected/68b388b0-9081-4e9f-9bc4-c0f4b91397fd-kube-api-access-2jfwc\") pod \"maas-default-gateway-openshift-default-6886b96f9-lt55h\" (UID: \"68b388b0-9081-4e9f-9bc4-c0f4b91397fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.236740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236705 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.236922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236747 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.236922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236781 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.236922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236835 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.236922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236862 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.236922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236896 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0369ac9e-dda1-4f8a-878a-24bddecac57b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.237202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236927 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5km\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-kube-api-access-nc5km\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.237202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.236965 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.237202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.237004 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.237350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.237224 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.237740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.237619 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.245196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.237707 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.245196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.238304 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0369ac9e-dda1-4f8a-878a-24bddecac57b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.245196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.238355 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.245196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.243789 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.245196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.244941 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.246247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.245846 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5km\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-kube-api-access-nc5km\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.250538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.250512 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gr8cg\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.325681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.325603 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:01.345383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.345355 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:01.613323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.613241 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h"] Nov 01 22:34:01.619170 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:01.619128 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b388b0_9081_4e9f_9bc4_c0f4b91397fd.slice/crio-ebfcee785e57379302b88e5ee6f2db75469f5b17714b9a71ac74a4a4521ff56e WatchSource:0}: Error finding container ebfcee785e57379302b88e5ee6f2db75469f5b17714b9a71ac74a4a4521ff56e: Status 404 returned error can't find the container with id ebfcee785e57379302b88e5ee6f2db75469f5b17714b9a71ac74a4a4521ff56e Nov 01 22:34:01.637830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:01.635993 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg"] Nov 01 22:34:01.638500 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:01.638471 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0369ac9e_dda1_4f8a_878a_24bddecac57b.slice/crio-42d589a0c3ff2c580038b06831f0432d13335f92b86f64d2448ee398ea4c7397 WatchSource:0}: Error finding container 42d589a0c3ff2c580038b06831f0432d13335f92b86f64d2448ee398ea4c7397: Status 404 returned error can't find the container with id 42d589a0c3ff2c580038b06831f0432d13335f92b86f64d2448ee398ea4c7397 Nov 01 22:34:02.237529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:02.237486 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" event={"ID":"0369ac9e-dda1-4f8a-878a-24bddecac57b","Type":"ContainerStarted","Data":"42d589a0c3ff2c580038b06831f0432d13335f92b86f64d2448ee398ea4c7397"} Nov 01 22:34:02.239136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:02.239107 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" event={"ID":"68b388b0-9081-4e9f-9bc4-c0f4b91397fd","Type":"ContainerStarted","Data":"ebfcee785e57379302b88e5ee6f2db75469f5b17714b9a71ac74a4a4521ff56e"} Nov 01 22:34:03.246215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:03.246180 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" event={"ID":"fd1ade81-64b5-4dcf-a5da-55632908568b","Type":"ContainerStarted","Data":"2c3954d9f5f008b6727d8c4abf78cd5ca79ec200706b1648a11229ef93945a7c"} Nov 01 22:34:03.246652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:03.246443 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:34:03.264380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:03.264322 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" podStartSLOduration=1.758746468 podStartE2EDuration="4.264305524s" totalImagesPullingTime="2.505559056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:03.262348045 +0000 UTC m=+608.690225977" watchObservedRunningTime="2026-11-01 22:34:03.264305524 +0000 UTC m=+608.692183454" Nov 01 22:34:04.256943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:04.256378 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" event={"ID":"d3deba2b-ac37-495d-bee9-e015774a9626","Type":"ContainerStarted","Data":"ed8b18af278b5ef1ee82443520f7dbc8a53a61239444d7b42f0b341cb6f2c74b"} Nov 01 22:34:04.256943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:04.256471 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:34:04.276032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:04.275976 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" podStartSLOduration=2.167971609 podStartE2EDuration="5.275961556s" totalImagesPullingTime="3.107989947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:04.275111833 +0000 UTC m=+609.702989764" watchObservedRunningTime="2026-11-01 22:34:04.275961556 +0000 UTC m=+609.703839486" Nov 01 22:34:05.560179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:05.560145 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:05.560547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:05.560232 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:05.560547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:05.560263 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:06.066475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.066437 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:06.066622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.066522 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:06.066622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.066569 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:06.269526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.269476 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" event={"ID":"68b388b0-9081-4e9f-9bc4-c0f4b91397fd","Type":"ContainerStarted","Data":"64761f0c021dab293800d577ad6d20d868c1aaa83a176f8d62c29d043604d2aa"} Nov 01 22:34:06.271335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.271303 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" event={"ID":"0369ac9e-dda1-4f8a-878a-24bddecac57b","Type":"ContainerStarted","Data":"e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d"} Nov 01 22:34:06.301394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.300849 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podStartSLOduration=1.857213738 podStartE2EDuration="6.300832381s" totalImagesPullingTime="4.443618643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:06.300136606 +0000 UTC m=+611.728014553" watchObservedRunningTime="2026-11-01 22:34:06.300832381 +0000 UTC m=+611.728710314" Nov 01 22:34:06.321231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.321118 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podStartSLOduration=1.402582979 podStartE2EDuration="5.321097329s" totalImagesPullingTime="3.91851435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:06.319123296 +0000 UTC m=+611.747001225" watchObservedRunningTime="2026-11-01 22:34:06.321097329 +0000 UTC m=+611.748975257" Nov 01 22:34:06.326259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.326231 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:06.346492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.346462 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:06.347855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.347819 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 22:34:06.347950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:06.347866 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 22:34:07.327443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:07.327403 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 22:34:07.327972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:07.327453 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 22:34:07.346085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:07.346056 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 22:34:07.346270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:07.346097 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 22:34:08.326224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:08.326174 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 22:34:08.326442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:08.326244 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 22:34:08.345765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:08.345738 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 22:34:08.346067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:08.345789 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 22:34:09.224603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.223911 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:34:09.327843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.327805 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 22:34:09.328065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.327877 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 22:34:09.346680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.346651 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 22:34:09.347063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.346711 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 22:34:09.444440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.444399 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-767cb84c4f-zld2b"] Nov 01 22:34:09.448404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.448386 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:09.457275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.457253 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-767cb84c4f-zld2b"] Nov 01 22:34:09.546349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.546261 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kf2b\" (UniqueName: \"kubernetes.io/projected/bc8e43f0-c141-4ba7-a4d1-cdc273722c58-kube-api-access-9kf2b\") pod \"ai-gateway-controller-767cb84c4f-zld2b\" (UID: \"bc8e43f0-c141-4ba7-a4d1-cdc273722c58\") " pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:09.546502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.546392 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8e43f0-c141-4ba7-a4d1-cdc273722c58-metrics-certs\") pod \"ai-gateway-controller-767cb84c4f-zld2b\" (UID: \"bc8e43f0-c141-4ba7-a4d1-cdc273722c58\") " pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:09.647825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.647787 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8e43f0-c141-4ba7-a4d1-cdc273722c58-metrics-certs\") pod \"ai-gateway-controller-767cb84c4f-zld2b\" (UID: \"bc8e43f0-c141-4ba7-a4d1-cdc273722c58\") " pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:09.648000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.647875 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kf2b\" (UniqueName: \"kubernetes.io/projected/bc8e43f0-c141-4ba7-a4d1-cdc273722c58-kube-api-access-9kf2b\") pod \"ai-gateway-controller-767cb84c4f-zld2b\" (UID: \"bc8e43f0-c141-4ba7-a4d1-cdc273722c58\") " pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:09.650300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.650278 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc8e43f0-c141-4ba7-a4d1-cdc273722c58-metrics-certs\") pod \"ai-gateway-controller-767cb84c4f-zld2b\" (UID: \"bc8e43f0-c141-4ba7-a4d1-cdc273722c58\") " pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:09.656264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.656231 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kf2b\" (UniqueName: \"kubernetes.io/projected/bc8e43f0-c141-4ba7-a4d1-cdc273722c58-kube-api-access-9kf2b\") pod \"ai-gateway-controller-767cb84c4f-zld2b\" (UID: \"bc8e43f0-c141-4ba7-a4d1-cdc273722c58\") " pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:09.763082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:09.763046 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:10.096599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:10.088510 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-767cb84c4f-zld2b"] Nov 01 22:34:10.126654 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:10.126618 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8e43f0_c141_4ba7_a4d1_cdc273722c58.slice/crio-b5481139e271d369890f9f9da3033d4b51362783a04bed19c86c9ab45bbd55b9 WatchSource:0}: Error finding container b5481139e271d369890f9f9da3033d4b51362783a04bed19c86c9ab45bbd55b9: Status 404 returned error can't find the container with id b5481139e271d369890f9f9da3033d4b51362783a04bed19c86c9ab45bbd55b9 Nov 01 22:34:10.326516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:10.326472 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 22:34:10.326714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:10.326536 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 22:34:10.332399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:10.332371 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" event={"ID":"bc8e43f0-c141-4ba7-a4d1-cdc273722c58","Type":"ContainerStarted","Data":"b5481139e271d369890f9f9da3033d4b51362783a04bed19c86c9ab45bbd55b9"} Nov 01 22:34:10.346381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:10.346351 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 22:34:10.346485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:10.346397 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 22:34:11.327044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:11.326999 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 22:34:11.327543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:11.327057 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 22:34:11.338061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:11.338028 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" event={"ID":"bc8e43f0-c141-4ba7-a4d1-cdc273722c58","Type":"ContainerStarted","Data":"14647e2579749c2e145978f6e1ed3a314e350640dff376edf12c9b9f5060c1d4"} Nov 01 22:34:11.338237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:11.338194 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:11.345905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:11.345877 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 22:34:11.346017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:11.345917 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 22:34:11.355324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:11.355281 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" podStartSLOduration=1.937437986 podStartE2EDuration="2.3552603s" totalImagesPullingTime="417.822314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:11.35334835 +0000 UTC m=+616.781226282" watchObservedRunningTime="2026-11-01 22:34:11.3552603 +0000 UTC m=+616.783138227" Nov 01 22:34:12.154729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:12.154681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:12.326502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:12.326467 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 22:34:12.326687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:12.326521 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 22:34:12.345784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:12.345740 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 01 22:34:12.346270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:12.345799 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 01 22:34:13.327047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:13.327004 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 01 22:34:13.327208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:13.327070 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 01 22:34:13.995753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:13.995720 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:13.996453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:13.996434 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:13.997493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:13.997472 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:34:14.260566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:14.260472 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" Nov 01 22:34:14.443736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:14.443705 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:14.444724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:14.444401 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:14.445214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:14.445198 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" Nov 01 22:34:14.524384 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:34:14.524305 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:34:15.265771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:15.265748 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-sph7r" Nov 01 22:34:16.046832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.046797 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr"] Nov 01 22:34:16.051390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.051367 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.073365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.073333 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr"] Nov 01 22:34:16.108593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108545 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zks\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-kube-api-access-78zks\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.108738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108662 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.108738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108703 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7cafbc-700c-43eb-871f-1427195d090d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.108809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108786 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.108846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108818 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7cafbc-700c-43eb-871f-1427195d090d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.108880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108844 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.108963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108947 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.109002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108977 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.109002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.108994 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209410 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zks\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-kube-api-access-78zks\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209458 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209482 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7cafbc-700c-43eb-871f-1427195d090d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209524 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209546 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7cafbc-700c-43eb-871f-1427195d090d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209597 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209647 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209680 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.209901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209823 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.210064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.209977 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.210064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.210020 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.210265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.210235 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.210385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.210300 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.210446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.210386 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7cafbc-700c-43eb-871f-1427195d090d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.211679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.211660 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.211994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.211977 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7cafbc-700c-43eb-871f-1427195d090d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.217485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.217459 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.217562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.217522 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zks\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-kube-api-access-78zks\") pod \"maas-default-gateway-openshift-default-6886b96f9-vmhfr\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.372558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.372489 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:16.649659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.649635 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr"] Nov 01 22:34:16.652547 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:16.652521 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7cafbc_700c_43eb_871f_1427195d090d.slice/crio-8453097ecad92ca97473b14d2c74f7946bfba705a9a67f70ae362d690921e444 WatchSource:0}: Error finding container 8453097ecad92ca97473b14d2c74f7946bfba705a9a67f70ae362d690921e444: Status 404 returned error can't find the container with id 8453097ecad92ca97473b14d2c74f7946bfba705a9a67f70ae362d690921e444 Nov 01 22:34:16.656127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.656097 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:16.656252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.656159 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:16.656252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:16.656194 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:17.379609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:17.378396 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" event={"ID":"1b7cafbc-700c-43eb-871f-1427195d090d","Type":"ContainerStarted","Data":"c643be0b55327451feb71ddb41d7e50f82e5db204c13b379df5f47cf6238a63e"} Nov 01 22:34:17.379609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:17.378471 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" event={"ID":"1b7cafbc-700c-43eb-871f-1427195d090d","Type":"ContainerStarted","Data":"8453097ecad92ca97473b14d2c74f7946bfba705a9a67f70ae362d690921e444"} Nov 01 22:34:17.407539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:17.407472 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podStartSLOduration=1.407452597 podStartE2EDuration="1.407452597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:17.405075046 +0000 UTC m=+622.832952979" watchObservedRunningTime="2026-11-01 22:34:17.407452597 +0000 UTC m=+622.835330530" Nov 01 22:34:18.372894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:18.372852 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:18.398738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:18.398702 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:34:18.399193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:18.398757 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:34:19.372926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:19.372887 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:34:19.373077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:19.372945 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:34:20.388767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:20.388730 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:34:20.389789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:20.389349 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:34:21.384599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:21.373856 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:34:21.384599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:21.373916 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:34:22.350889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.350859 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-767cb84c4f-zld2b" Nov 01 22:34:22.373010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.372981 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:34:22.373163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.373027 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:34:22.400592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.400537 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-h8248"] Nov 01 22:34:22.400857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.400801 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" podUID="b7374fdc-9e86-410d-a9aa-3579563600b8" containerName="manager" containerID="cri-o://8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08" gracePeriod=30 Nov 01 22:34:22.652176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.652152 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:34:22.773355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.773328 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7374fdc-9e86-410d-a9aa-3579563600b8-kube-api-access-7tnbf\" (UniqueName: \"kubernetes.io/projected/b7374fdc-9e86-410d-a9aa-3579563600b8-kube-api-access-7tnbf\") pod \"b7374fdc-9e86-410d-a9aa-3579563600b8\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " Nov 01 22:34:22.773495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.773383 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7374fdc-9e86-410d-a9aa-3579563600b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7374fdc-9e86-410d-a9aa-3579563600b8-metrics-certs\") pod \"b7374fdc-9e86-410d-a9aa-3579563600b8\" (UID: \"b7374fdc-9e86-410d-a9aa-3579563600b8\") " Nov 01 22:34:22.775594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.775559 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7374fdc-9e86-410d-a9aa-3579563600b8-kube-api-access-7tnbf" pod "b7374fdc-9e86-410d-a9aa-3579563600b8" (UID: "b7374fdc-9e86-410d-a9aa-3579563600b8"). InnerVolumeSpecName "kube-api-access-7tnbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:34:22.775856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.775841 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7374fdc-9e86-410d-a9aa-3579563600b8-metrics-certs" pod "b7374fdc-9e86-410d-a9aa-3579563600b8" (UID: "b7374fdc-9e86-410d-a9aa-3579563600b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:34:22.874693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.874663 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tnbf\" (UniqueName: \"kubernetes.io/projected/b7374fdc-9e86-410d-a9aa-3579563600b8-kube-api-access-7tnbf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:22.874693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:22.874690 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7374fdc-9e86-410d-a9aa-3579563600b8-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:34:23.373816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.373778 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 01 22:34:23.374263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.373836 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 01 22:34:23.428277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.428241 4981 generic.go:356] "Generic (PLEG): container finished" podID="b7374fdc-9e86-410d-a9aa-3579563600b8" containerID="8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08" exitCode=0 Nov 01 22:34:23.428438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.428285 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" Nov 01 22:34:23.428438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.428327 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" event={"ID":"b7374fdc-9e86-410d-a9aa-3579563600b8","Type":"ContainerDied","Data":"8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08"} Nov 01 22:34:23.428438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.428382 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-h8248" event={"ID":"b7374fdc-9e86-410d-a9aa-3579563600b8","Type":"ContainerDied","Data":"e6a2415b7fc0562a187abaabc1f98da2002d316e3468027c6eee26369b31556b"} Nov 01 22:34:23.428438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.428409 4981 scope.go:122] "RemoveContainer" containerID="8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08" Nov 01 22:34:23.438112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.438097 4981 scope.go:122] "RemoveContainer" containerID="8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08" Nov 01 22:34:23.438378 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:34:23.438360 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08\": container with ID starting with 8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08 not found: ID does not exist" containerID="8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08" Nov 01 22:34:23.438431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.438385 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08"} err="failed to get container status \"8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08\": rpc error: code = NotFound desc = could not find container \"8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08\": container with ID starting with 8467e2af9c7b546eff076e614ed4eb5a2f72ffcb2ca2a7083bbc0e7cae665a08 not found: ID does not exist" Nov 01 22:34:23.450080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.450048 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-h8248"] Nov 01 22:34:23.453291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.453269 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-h8248"] Nov 01 22:34:23.913470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:23.913435 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:24.370032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:24.370000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:24.732199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:24.732167 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7374fdc-9e86-410d-a9aa-3579563600b8" path="/var/lib/kubelet/pods/b7374fdc-9e86-410d-a9aa-3579563600b8/volumes" Nov 01 22:34:24.855367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:24.855328 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:24.855765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:24.855743 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:24.856565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:24.856539 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:34:25.906326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:25.906296 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:26.009396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:26.009360 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:27.237416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:27.237373 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:27.368675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:27.368640 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:31.068425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.068387 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc"] Nov 01 22:34:31.069064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.069040 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b7374fdc-9e86-410d-a9aa-3579563600b8" containerName="manager" Nov 01 22:34:31.069193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.069067 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7374fdc-9e86-410d-a9aa-3579563600b8" containerName="manager" Nov 01 22:34:31.069193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.069185 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7374fdc-9e86-410d-a9aa-3579563600b8" containerName="manager" Nov 01 22:34:31.075900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.075879 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.085902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.085877 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc"] Nov 01 22:34:31.156121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156084 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156133 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156225 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156275 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156345 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5fc\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-kube-api-access-gs5fc\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156457 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156499 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156569 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.156681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.156636 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258030 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258075 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258100 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258120 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258139 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258168 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5fc\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-kube-api-access-gs5fc\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258229 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258264 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258299 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258476 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258508 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258722 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258749 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.258915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.258868 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.260620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.260598 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.260694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.260645 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.267482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.267458 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5fc\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-kube-api-access-gs5fc\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.268068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.268050 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kxkqc\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.393522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.393444 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:31.672790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.672760 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc"] Nov 01 22:34:31.674973 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:31.674942 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34fccec_9a46_4bc1_87ed_5de63a8b0a1c.slice/crio-e4669a6b9702150f4eb73ee7e05c4d25bd4557e13fd8afdaad114574556b37ec WatchSource:0}: Error finding container e4669a6b9702150f4eb73ee7e05c4d25bd4557e13fd8afdaad114574556b37ec: Status 404 returned error can't find the container with id e4669a6b9702150f4eb73ee7e05c4d25bd4557e13fd8afdaad114574556b37ec Nov 01 22:34:31.677284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.677252 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:31.677380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.677332 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:31.677380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:31.677374 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:32.490599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:32.473677 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" event={"ID":"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c","Type":"ContainerStarted","Data":"095e97bce8e545776f55c677c6f8f31f08f48a1e11ff8b88fb3c889c40cc3b45"} Nov 01 22:34:32.490599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:32.473739 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" event={"ID":"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c","Type":"ContainerStarted","Data":"e4669a6b9702150f4eb73ee7e05c4d25bd4557e13fd8afdaad114574556b37ec"} Nov 01 22:34:32.523933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:32.523883 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podStartSLOduration=1.523868228 podStartE2EDuration="1.523868228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:32.522006346 +0000 UTC m=+637.949884266" watchObservedRunningTime="2026-11-01 22:34:32.523868228 +0000 UTC m=+637.951746159" Nov 01 22:34:33.398603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:33.397872 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:33.399739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:33.399713 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:34:33.399875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:33.399759 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:34:34.394044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:34.394004 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:34:34.394667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:34.394635 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:34:35.399049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:35.398961 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:34:35.399049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:35.399030 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:34:36.394981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:36.394941 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:34:36.395286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:36.395259 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:34:37.398617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:37.397974 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:34:37.398617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:37.398057 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:34:38.393823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:38.393791 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 22:34:38.393983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:38.393841 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 22:34:39.838904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:39.838867 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:39.839390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:39.839174 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:39.840290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:39.840269 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 22:34:44.522825 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:34:44.522794 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:34:52.108921 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.108885 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:52.190077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.190048 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:52.559872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.559843 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:52.611853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.611824 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm"] Nov 01 22:34:52.615599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.615553 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.624353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.624335 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2f3705-openshift-default-dockercfg-29c4c\"" Nov 01 22:34:52.633109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.633086 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm"] Nov 01 22:34:52.680714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.680682 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.680852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.680724 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.680852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.680750 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6sd8\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-kube-api-access-s6sd8\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.680955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.680896 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.680955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.680937 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.681027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.681004 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.681154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.681043 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.681154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.681061 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.681154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.681125 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.701696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.701672 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:52.781952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.781912 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782002 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782047 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782086 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782112 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782147 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782212 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782260 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782299 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6sd8\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-kube-api-access-s6sd8\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782461 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782470 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782526 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782745 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.782839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.782818 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.784813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.784786 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.784974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.784951 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.799315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.799246 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6sd8\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-kube-api-access-s6sd8\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.805435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.803378 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:52.875409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.875328 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:52.885479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.885451 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz"] Nov 01 22:34:52.890982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.890962 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:52.900110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.900092 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-927b68-openshift-default-dockercfg-rfvvh\"" Nov 01 22:34:52.910491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.910468 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz"] Nov 01 22:34:52.931088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:52.931059 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:53.035599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.035534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6182a7fa-5369-41de-9122-45ca7836627d-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.035758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.035706 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6182a7fa-5369-41de-9122-45ca7836627d-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.035801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.035759 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.035841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.035806 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.035932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.035914 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.035998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.035983 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.036033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.036010 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk66t\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-kube-api-access-jk66t\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.036105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.036090 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.036173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.036128 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.136934 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6182a7fa-5369-41de-9122-45ca7836627d-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.136965 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6182a7fa-5369-41de-9122-45ca7836627d-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.136993 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137027 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137096 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137143 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137169 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk66t\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-kube-api-access-jk66t\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137212 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137240 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137496 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137594 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137614 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137742 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6182a7fa-5369-41de-9122-45ca7836627d-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.137836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.137744 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.139401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.139381 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6182a7fa-5369-41de-9122-45ca7836627d-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.139501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.139459 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.175312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.175286 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.175612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.175592 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk66t\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-kube-api-access-jk66t\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.201208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.201182 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:53.206130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.206110 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:53.234131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.234022 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm"] Nov 01 22:34:53.245044 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:53.245002 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e14a2a1_e3bd_4bb4_a477_06d73f87df06.slice/crio-849d1385fa644d395680ea024bae14557343aac5bed7b04df3476fd9141e1935 WatchSource:0}: Error finding container 849d1385fa644d395680ea024bae14557343aac5bed7b04df3476fd9141e1935: Status 404 returned error can't find the container with id 849d1385fa644d395680ea024bae14557343aac5bed7b04df3476fd9141e1935 Nov 01 22:34:53.247102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.247058 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:53.247102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.247145 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:53.247380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.247185 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:53.370674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.370644 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2"] Nov 01 22:34:53.375095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.375071 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.395365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.395290 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9ad303-openshift-default-dockercfg-cbwmc\"" Nov 01 22:34:53.413118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.413057 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2"] Nov 01 22:34:53.487717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.487649 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:53.540402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.540366 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz"] Nov 01 22:34:53.540732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.540709 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.540845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.540746 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgt4f\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-kube-api-access-cgt4f\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.540895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.540867 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.540937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.540909 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.540972 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:53.540935 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6182a7fa_5369_41de_9122_45ca7836627d.slice/crio-c1337a4d2cfc8cb37555053bb9b0a17bf2e72cc1203a22c663b2c16151b96148 WatchSource:0}: Error finding container c1337a4d2cfc8cb37555053bb9b0a17bf2e72cc1203a22c663b2c16151b96148: Status 404 returned error can't find the container with id c1337a4d2cfc8cb37555053bb9b0a17bf2e72cc1203a22c663b2c16151b96148 Nov 01 22:34:53.541048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.541032 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.541092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.541067 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.541140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.541125 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.541235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.541193 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad33ddb9-b1b9-4b1f-9691-607def69c690-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.541235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.541224 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.542834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.542799 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:53.542947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.542875 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:53.542947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.542929 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:53.600356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.600322 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" event={"ID":"1e14a2a1-e3bd-4bb4-a477-06d73f87df06","Type":"ContainerStarted","Data":"815bba694e5437fd06c34f709cf97999d5fc90151a8d55762db0ef48f430e587"} Nov 01 22:34:53.600491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.600386 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" event={"ID":"1e14a2a1-e3bd-4bb4-a477-06d73f87df06","Type":"ContainerStarted","Data":"849d1385fa644d395680ea024bae14557343aac5bed7b04df3476fd9141e1935"} Nov 01 22:34:53.603163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.603139 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" event={"ID":"6182a7fa-5369-41de-9122-45ca7836627d","Type":"ContainerStarted","Data":"c1337a4d2cfc8cb37555053bb9b0a17bf2e72cc1203a22c663b2c16151b96148"} Nov 01 22:34:53.615681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.615656 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb"] Nov 01 22:34:53.619864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.619841 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.628108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.628088 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f338e4-openshift-default-dockercfg-ksfnw\"" Nov 01 22:34:53.642007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.641974 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642017 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642054 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642123 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad33ddb9-b1b9-4b1f-9691-607def69c690-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642162 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642235 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642260 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgt4f\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-kube-api-access-cgt4f\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642322 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642351 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642668 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.642973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.642788 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.643196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.643018 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad33ddb9-b1b9-4b1f-9691-607def69c690-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.643311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.643234 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.643436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.643389 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.645768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.645502 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podStartSLOduration=1.645490094 podStartE2EDuration="1.645490094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:53.643419426 +0000 UTC m=+659.071297358" watchObservedRunningTime="2026-11-01 22:34:53.645490094 +0000 UTC m=+659.073368024" Nov 01 22:34:53.647206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.646274 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.647881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.647831 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.651295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.651272 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb"] Nov 01 22:34:53.676263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.676136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:53.677850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.677828 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.678143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.678122 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgt4f\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-kube-api-access-cgt4f\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.693803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.693777 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:53.743787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.743751 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.743964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.743831 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.743964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.743902 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c13c7e4-17a5-470b-becd-8c18896915d3-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.744199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.743990 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.744199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.744105 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.744329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.744185 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.744329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.744239 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.744329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.744321 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.744627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.744376 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkbn\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-kube-api-access-pjkbn\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.845624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845589 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.845819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845642 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.845819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845666 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkbn\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-kube-api-access-pjkbn\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.845819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845724 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.845819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845755 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.845819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845818 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c13c7e4-17a5-470b-becd-8c18896915d3-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.846098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845876 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.846098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.845947 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.846098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.846013 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.846483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.846443 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.846483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.846474 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.846895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.846877 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.847026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.846938 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c13c7e4-17a5-470b-becd-8c18896915d3-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.847026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.846954 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.848098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.848077 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.848324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.848303 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.862984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.862956 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.864538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.864503 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkbn\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-kube-api-access-pjkbn\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.931362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.931337 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:53.934632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.934567 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:53.976089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:53.976057 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:54.029269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.029186 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2"] Nov 01 22:34:54.031835 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:54.031801 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad33ddb9_b1b9_4b1f_9691_607def69c690.slice/crio-74e4fee70b43f0728360b6fe5d4de36a6c42568e163a9ad6d82687ea4668efe9 WatchSource:0}: Error finding container 74e4fee70b43f0728360b6fe5d4de36a6c42568e163a9ad6d82687ea4668efe9: Status 404 returned error can't find the container with id 74e4fee70b43f0728360b6fe5d4de36a6c42568e163a9ad6d82687ea4668efe9 Nov 01 22:34:54.034783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.034749 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:54.034888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.034826 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:54.034888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.034867 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:54.220624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.220599 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb"] Nov 01 22:34:54.223249 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:54.223223 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c13c7e4_17a5_470b_becd_8c18896915d3.slice/crio-646901fa50f1a0658182a2420f60fa1f4a8949d49586e3ab08e355859357f91c WatchSource:0}: Error finding container 646901fa50f1a0658182a2420f60fa1f4a8949d49586e3ab08e355859357f91c: Status 404 returned error can't find the container with id 646901fa50f1a0658182a2420f60fa1f4a8949d49586e3ab08e355859357f91c Nov 01 22:34:54.226150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.226117 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:54.226261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.226195 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:54.226261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.226226 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:34:54.609545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.609455 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" event={"ID":"6182a7fa-5369-41de-9122-45ca7836627d","Type":"ContainerStarted","Data":"9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6"} Nov 01 22:34:54.611098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.611074 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" event={"ID":"ad33ddb9-b1b9-4b1f-9691-607def69c690","Type":"ContainerStarted","Data":"d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a"} Nov 01 22:34:54.611234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.611115 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" event={"ID":"ad33ddb9-b1b9-4b1f-9691-607def69c690","Type":"ContainerStarted","Data":"74e4fee70b43f0728360b6fe5d4de36a6c42568e163a9ad6d82687ea4668efe9"} Nov 01 22:34:54.612605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.612559 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" event={"ID":"3c13c7e4-17a5-470b-becd-8c18896915d3","Type":"ContainerStarted","Data":"7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008"} Nov 01 22:34:54.612701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.612626 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" event={"ID":"3c13c7e4-17a5-470b-becd-8c18896915d3","Type":"ContainerStarted","Data":"646901fa50f1a0658182a2420f60fa1f4a8949d49586e3ab08e355859357f91c"} Nov 01 22:34:54.635692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.635648 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podStartSLOduration=2.63563277 podStartE2EDuration="2.63563277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:54.635362155 +0000 UTC m=+660.063240087" watchObservedRunningTime="2026-11-01 22:34:54.63563277 +0000 UTC m=+660.063510700" Nov 01 22:34:54.662807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.662768 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podStartSLOduration=1.662750875 podStartE2EDuration="1.662750875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:54.662162469 +0000 UTC m=+660.090040405" watchObservedRunningTime="2026-11-01 22:34:54.662750875 +0000 UTC m=+660.090628805" Nov 01 22:34:54.687387 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.687330 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podStartSLOduration=1.687314741 podStartE2EDuration="1.687314741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:54.685127272 +0000 UTC m=+660.113005203" watchObservedRunningTime="2026-11-01 22:34:54.687314741 +0000 UTC m=+660.115192674" Nov 01 22:34:54.694189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.694169 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:54.935484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.935458 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:54.936885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:54.936864 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:55.206594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.206475 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:55.211440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.211418 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:55.618420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.618322 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:55.618896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.618501 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:55.619128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.619107 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:34:55.619252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.619242 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:34:55.700433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.700400 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:55.940458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.940433 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:55.979567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:55.979540 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:56.276933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.276856 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:56.477137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.477104 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:56.618363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.618278 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:56.622700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.622665 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:56.623075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.622716 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:56.623220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.623205 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:34:56.623665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.623648 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:34:56.940239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:56.940159 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:57.128669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:57.128625 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:57.471506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:57.471473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:57.736739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:57.736665 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:57.940233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:57.940199 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:58.068237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:58.068170 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:59.014094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.014059 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:59.078314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.078280 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk"] Nov 01 22:34:59.083312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.083293 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.097173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.097154 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e26733-dockercfg-26fj9\"" Nov 01 22:34:59.102769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.102746 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa48762-a511-4b63-b3c1-cf90fc387cb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.102874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.102852 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nw67\" (UniqueName: \"kubernetes.io/projected/cfa48762-a511-4b63-b3c1-cf90fc387cb4-kube-api-access-6nw67\") pod \"payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.103536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.103519 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e26733\"" Nov 01 22:34:59.108071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.108050 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk"] Nov 01 22:34:59.117343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.117312 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c"] Nov 01 22:34:59.120964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.120945 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.137317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.137291 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c"] Nov 01 22:34:59.222321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.222281 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nw67\" (UniqueName: \"kubernetes.io/projected/cfa48762-a511-4b63-b3c1-cf90fc387cb4-kube-api-access-6nw67\") pod \"payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.222504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.222438 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.222617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.222593 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa48762-a511-4b63-b3c1-cf90fc387cb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.222718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.222673 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2klp\" (UniqueName: \"kubernetes.io/projected/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-kube-api-access-g2klp\") pod \"payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.222718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.222674 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:59.224307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.223901 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa48762-a511-4b63-b3c1-cf90fc387cb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.235763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.235362 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nw67\" (UniqueName: \"kubernetes.io/projected/cfa48762-a511-4b63-b3c1-cf90fc387cb4-kube-api-access-6nw67\") pod \"payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.323876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.323810 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2klp\" (UniqueName: \"kubernetes.io/projected/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-kube-api-access-g2klp\") pod \"payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.324007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.323906 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.324515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.324497 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.345985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.345955 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2klp\" (UniqueName: \"kubernetes.io/projected/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-kube-api-access-g2klp\") pod \"payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.395886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.395858 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:34:59.435421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.435387 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:34:59.537895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.537867 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:59.611615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.611527 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2"] Nov 01 22:34:59.616136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.616113 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:34:59.623929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.623906 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e5f47e\"" Nov 01 22:34:59.624022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.623914 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e5f47e-dockercfg-xhwjj\"" Nov 01 22:34:59.626805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.626776 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdfcm\" (UniqueName: \"kubernetes.io/projected/f58e78c4-ffa9-4957-985d-682836f68224-kube-api-access-jdfcm\") pod \"payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:34:59.627008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.626983 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e78c4-ffa9-4957-985d-682836f68224-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:34:59.629059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.629037 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2"] Nov 01 22:34:59.687645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.687609 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk"] Nov 01 22:34:59.688205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.688176 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:59.689624 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:59.689598 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa48762_a511_4b63_b3c1_cf90fc387cb4.slice/crio-98dad3c59e4803fab11ef1b32394bff7200ae1f90ee528a3ba328efa55a47015 WatchSource:0}: Error finding container 98dad3c59e4803fab11ef1b32394bff7200ae1f90ee528a3ba328efa55a47015: Status 404 returned error can't find the container with id 98dad3c59e4803fab11ef1b32394bff7200ae1f90ee528a3ba328efa55a47015 Nov 01 22:34:59.737645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.732007 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfcm\" (UniqueName: \"kubernetes.io/projected/f58e78c4-ffa9-4957-985d-682836f68224-kube-api-access-jdfcm\") pod \"payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:34:59.737645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.732108 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e78c4-ffa9-4957-985d-682836f68224-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:34:59.737645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.733082 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e78c4-ffa9-4957-985d-682836f68224-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:34:59.745493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.745451 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c"] Nov 01 22:34:59.750881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.750844 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfcm\" (UniqueName: \"kubernetes.io/projected/f58e78c4-ffa9-4957-985d-682836f68224-kube-api-access-jdfcm\") pod \"payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:34:59.757313 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:34:59.757269 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2028d1a7_5b1d_4c6c_b1da_a53c877ad27c.slice/crio-d8fb48bf21dc994a2f54629c13f52bf8df9b5a855fa616f63937ab7b5b11a0dd WatchSource:0}: Error finding container d8fb48bf21dc994a2f54629c13f52bf8df9b5a855fa616f63937ab7b5b11a0dd: Status 404 returned error can't find the container with id d8fb48bf21dc994a2f54629c13f52bf8df9b5a855fa616f63937ab7b5b11a0dd Nov 01 22:34:59.906559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.906469 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:34:59.930735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:34:59.930703 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:35:00.016563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.016526 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:00.226331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.226298 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2"] Nov 01 22:35:00.230326 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:00.230301 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58e78c4_ffa9_4957_985d_682836f68224.slice/crio-fa069e642f1889a847624e55976bcffd87adf92d775abc66e4d17830dde48254 WatchSource:0}: Error finding container fa069e642f1889a847624e55976bcffd87adf92d775abc66e4d17830dde48254: Status 404 returned error can't find the container with id fa069e642f1889a847624e55976bcffd87adf92d775abc66e4d17830dde48254 Nov 01 22:35:00.645005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.644918 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" event={"ID":"f58e78c4-ffa9-4957-985d-682836f68224","Type":"ContainerStarted","Data":"a6c7f389cd73760f8cd5526ee216ca8967ba9b2b09100c9c0eacf30e1505e367"} Nov 01 22:35:00.645005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.644970 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" event={"ID":"f58e78c4-ffa9-4957-985d-682836f68224","Type":"ContainerStarted","Data":"fa069e642f1889a847624e55976bcffd87adf92d775abc66e4d17830dde48254"} Nov 01 22:35:00.645218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.645024 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:35:00.646717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.646681 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" event={"ID":"cfa48762-a511-4b63-b3c1-cf90fc387cb4","Type":"ContainerStarted","Data":"4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5"} Nov 01 22:35:00.646867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.646740 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" event={"ID":"cfa48762-a511-4b63-b3c1-cf90fc387cb4","Type":"ContainerStarted","Data":"98dad3c59e4803fab11ef1b32394bff7200ae1f90ee528a3ba328efa55a47015"} Nov 01 22:35:00.646867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.646769 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:35:00.648221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.648203 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" event={"ID":"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c","Type":"ContainerStarted","Data":"056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657"} Nov 01 22:35:00.648314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.648235 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" event={"ID":"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c","Type":"ContainerStarted","Data":"d8fb48bf21dc994a2f54629c13f52bf8df9b5a855fa616f63937ab7b5b11a0dd"} Nov 01 22:35:00.648351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.648327 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:35:00.715986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.715939 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" podStartSLOduration=1.715926975 podStartE2EDuration="1.715926975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:00.714845733 +0000 UTC m=+666.142723664" watchObservedRunningTime="2026-11-01 22:35:00.715926975 +0000 UTC m=+666.143804905" Nov 01 22:35:00.716129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.716030 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" podStartSLOduration=1.716023797 podStartE2EDuration="1.716023797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:00.686531144 +0000 UTC m=+666.114409075" watchObservedRunningTime="2026-11-01 22:35:00.716023797 +0000 UTC m=+666.143901733" Nov 01 22:35:00.749106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.749062 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" podStartSLOduration=1.749047625 podStartE2EDuration="1.749047625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:00.747502027 +0000 UTC m=+666.175379957" watchObservedRunningTime="2026-11-01 22:35:00.749047625 +0000 UTC m=+666.176925605" Nov 01 22:35:00.850788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:00.850749 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:01.016551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.016506 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:01.525919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.525888 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:01.632852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.632820 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2"] Nov 01 22:35:01.636824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.636802 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.646862 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.646830 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9ad303-dockercfg-2k8qk\"" Nov 01 22:35:01.650763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.649568 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9ad303\"" Nov 01 22:35:01.653127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.653107 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2"] Nov 01 22:35:01.664051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.664019 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf"] Nov 01 22:35:01.670249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.669052 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:01.683604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.683504 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf"] Nov 01 22:35:01.690139 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.690097 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:01.757279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.757236 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8d6\" (UniqueName: \"kubernetes.io/projected/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-kube-api-access-8s8d6\") pod \"payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.757463 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.757301 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.865770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.864825 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8d6\" (UniqueName: \"kubernetes.io/projected/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-kube-api-access-8s8d6\") pod \"payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.865770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.864886 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.865770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.865057 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babb7476-eab9-41b5-9729-c94dd542ad9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:01.865770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.865100 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264mj\" (UniqueName: \"kubernetes.io/projected/babb7476-eab9-41b5-9729-c94dd542ad9d-kube-api-access-264mj\") pod \"payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:01.869267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.869212 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.877980 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.877746 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8d6\" (UniqueName: \"kubernetes.io/projected/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-kube-api-access-8s8d6\") pod \"payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.950541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.950509 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:01.965992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.965956 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babb7476-eab9-41b5-9729-c94dd542ad9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:01.966119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.966008 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264mj\" (UniqueName: \"kubernetes.io/projected/babb7476-eab9-41b5-9729-c94dd542ad9d-kube-api-access-264mj\") pod \"payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:01.966745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.966722 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babb7476-eab9-41b5-9729-c94dd542ad9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:01.983245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.983211 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264mj\" (UniqueName: \"kubernetes.io/projected/babb7476-eab9-41b5-9729-c94dd542ad9d-kube-api-access-264mj\") pod \"payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:01.990839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:01.990805 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:02.005067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.005039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:02.100900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.100865 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c"] Nov 01 22:35:02.108034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.108001 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.128941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.128911 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c"] Nov 01 22:35:02.145476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.145448 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-6ac182-dockercfg-fdj7k\"" Nov 01 22:35:02.162467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.162431 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:02.208877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.203661 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-6ac182\"" Nov 01 22:35:02.260230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.260200 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2"] Nov 01 22:35:02.261738 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:02.261709 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a95663_85ce_4cee_a4ac_b5c6ec90398c.slice/crio-61234577382c3779852e64bf3e42182d4eafe05d018510d39282778f66d59cee WatchSource:0}: Error finding container 61234577382c3779852e64bf3e42182d4eafe05d018510d39282778f66d59cee: Status 404 returned error can't find the container with id 61234577382c3779852e64bf3e42182d4eafe05d018510d39282778f66d59cee Nov 01 22:35:02.270139 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.270115 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlsd\" (UniqueName: \"kubernetes.io/projected/d815af03-1261-4769-bf33-6a76cf1f17bd-kube-api-access-9vlsd\") pod \"payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.270238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.270176 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d815af03-1261-4769-bf33-6a76cf1f17bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.296064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.295814 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf"] Nov 01 22:35:02.302875 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:02.302841 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabb7476_eab9_41b5_9729_c94dd542ad9d.slice/crio-8cbb318cb62f448b92f059031b606c2532f0f5efe0ce8c2050ebbb55592b3393 WatchSource:0}: Error finding container 8cbb318cb62f448b92f059031b606c2532f0f5efe0ce8c2050ebbb55592b3393: Status 404 returned error can't find the container with id 8cbb318cb62f448b92f059031b606c2532f0f5efe0ce8c2050ebbb55592b3393 Nov 01 22:35:02.371515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.371490 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlsd\" (UniqueName: \"kubernetes.io/projected/d815af03-1261-4769-bf33-6a76cf1f17bd-kube-api-access-9vlsd\") pod \"payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.371599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.371557 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d815af03-1261-4769-bf33-6a76cf1f17bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.372471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.372447 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d815af03-1261-4769-bf33-6a76cf1f17bd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.381116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.381090 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlsd\" (UniqueName: \"kubernetes.io/projected/d815af03-1261-4769-bf33-6a76cf1f17bd-kube-api-access-9vlsd\") pod \"payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.423923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.423886 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:02.662480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.662381 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" event={"ID":"babb7476-eab9-41b5-9729-c94dd542ad9d","Type":"ContainerStarted","Data":"d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c"} Nov 01 22:35:02.662480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.662445 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" event={"ID":"babb7476-eab9-41b5-9729-c94dd542ad9d","Type":"ContainerStarted","Data":"8cbb318cb62f448b92f059031b606c2532f0f5efe0ce8c2050ebbb55592b3393"} Nov 01 22:35:02.663108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.662550 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:02.664119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.664094 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" event={"ID":"e8a95663-85ce-4cee-a4ac-b5c6ec90398c","Type":"ContainerStarted","Data":"f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01"} Nov 01 22:35:02.664223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.664136 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" event={"ID":"e8a95663-85ce-4cee-a4ac-b5c6ec90398c","Type":"ContainerStarted","Data":"61234577382c3779852e64bf3e42182d4eafe05d018510d39282778f66d59cee"} Nov 01 22:35:02.664223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.664155 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:02.691199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.691166 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:02.692929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.692881 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" podStartSLOduration=1.692867391 podStartE2EDuration="1.692867391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:02.688915258 +0000 UTC m=+668.116793190" watchObservedRunningTime="2026-11-01 22:35:02.692867391 +0000 UTC m=+668.120745322" Nov 01 22:35:02.715701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.715677 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c"] Nov 01 22:35:02.718086 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:02.718055 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd815af03_1261_4769_bf33_6a76cf1f17bd.slice/crio-991172c4a292d0604b56a2d955d9fb40a36a5a923e12bedef7b3f898eaa22815 WatchSource:0}: Error finding container 991172c4a292d0604b56a2d955d9fb40a36a5a923e12bedef7b3f898eaa22815: Status 404 returned error can't find the container with id 991172c4a292d0604b56a2d955d9fb40a36a5a923e12bedef7b3f898eaa22815 Nov 01 22:35:02.718784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.718742 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" podStartSLOduration=1.718729583 podStartE2EDuration="1.718729583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:02.716707532 +0000 UTC m=+668.144585465" watchObservedRunningTime="2026-11-01 22:35:02.718729583 +0000 UTC m=+668.146607519" Nov 01 22:35:02.799112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.799084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:02.862692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.862658 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn"] Nov 01 22:35:02.866521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.866502 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:02.883953 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.880512 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn"] Nov 01 22:35:02.936813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.936782 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:02.946186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.946165 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-f338e4-dockercfg-vf42d\"" Nov 01 22:35:02.949414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.949396 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-f338e4\"" Nov 01 22:35:02.979054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.979021 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtf8q\" (UniqueName: \"kubernetes.io/projected/42a1952a-0a95-40f4-8a94-d7580b97f74c-kube-api-access-dtf8q\") pod \"payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:02.979206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:02.979164 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a1952a-0a95-40f4-8a94-d7580b97f74c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:03.079993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.079955 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtf8q\" (UniqueName: \"kubernetes.io/projected/42a1952a-0a95-40f4-8a94-d7580b97f74c-kube-api-access-dtf8q\") pod \"payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:03.080201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.080087 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a1952a-0a95-40f4-8a94-d7580b97f74c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:03.080856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.080836 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a1952a-0a95-40f4-8a94-d7580b97f74c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:03.095729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.095704 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtf8q\" (UniqueName: \"kubernetes.io/projected/42a1952a-0a95-40f4-8a94-d7580b97f74c-kube-api-access-dtf8q\") pod \"payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:03.201297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.201218 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:03.479053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.479026 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn"] Nov 01 22:35:03.480892 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:03.480856 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a1952a_0a95_40f4_8a94_d7580b97f74c.slice/crio-484287c36267148270e7b55a31c0da1634c82e1acf19d76eeb8392d1149e0250 WatchSource:0}: Error finding container 484287c36267148270e7b55a31c0da1634c82e1acf19d76eeb8392d1149e0250: Status 404 returned error can't find the container with id 484287c36267148270e7b55a31c0da1634c82e1acf19d76eeb8392d1149e0250 Nov 01 22:35:03.673104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.673039 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" event={"ID":"42a1952a-0a95-40f4-8a94-d7580b97f74c","Type":"ContainerStarted","Data":"505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7"} Nov 01 22:35:03.673609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.673170 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" event={"ID":"42a1952a-0a95-40f4-8a94-d7580b97f74c","Type":"ContainerStarted","Data":"484287c36267148270e7b55a31c0da1634c82e1acf19d76eeb8392d1149e0250"} Nov 01 22:35:03.673609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.673425 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:03.677322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.674762 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:03.677322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.676873 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" event={"ID":"d815af03-1261-4769-bf33-6a76cf1f17bd","Type":"ContainerStarted","Data":"4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef"} Nov 01 22:35:03.677322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.676923 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" event={"ID":"d815af03-1261-4769-bf33-6a76cf1f17bd","Type":"ContainerStarted","Data":"991172c4a292d0604b56a2d955d9fb40a36a5a923e12bedef7b3f898eaa22815"} Nov 01 22:35:03.677322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.676960 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:03.699612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.699548 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" podStartSLOduration=1.699537813 podStartE2EDuration="1.699537813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:03.69547837 +0000 UTC m=+669.123356301" watchObservedRunningTime="2026-11-01 22:35:03.699537813 +0000 UTC m=+669.127415743" Nov 01 22:35:03.718667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:03.718619 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" podStartSLOduration=1.718605073 podStartE2EDuration="1.718605073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:03.716514835 +0000 UTC m=+669.144392768" watchObservedRunningTime="2026-11-01 22:35:03.718605073 +0000 UTC m=+669.146483001" Nov 01 22:35:04.689042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:04.689009 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:04.836857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:04.836825 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:04.948907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:04.948820 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:05.616551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:05.616515 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:05.752353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:05.752325 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:05.891650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:05.887360 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:06.603213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:06.603156 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:06.700078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:06.700031 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:06.799412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:06.799376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:07.395599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:07.394653 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:08.299280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:08.299233 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:08.396103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:08.396069 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:09.897716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:09.897653 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:10.864894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:10.864803 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:10.967626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:10.967551 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:11.497600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:11.492344 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:11.659154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:11.658571 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 22:35:11.659154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:11.659101 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 22:35:11.659615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:11.659593 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 22:35:11.660687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:11.660667 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:11.667701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:11.667680 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:35:11.667769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:11.667729 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:35:12.159600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:12.156970 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:12.353497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:12.353468 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:13.269680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.269636 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:13.450599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.442044 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8"] Nov 01 22:35:13.463178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.463145 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.465015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.464987 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8"] Nov 01 22:35:13.518626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.492839 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"kube-root-ca.crt\"" Nov 01 22:35:13.518626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.492921 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"openshift-service-ca.crt\"" Nov 01 22:35:13.518626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.493029 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"default-dockercfg-ghrq6\"" Nov 01 22:35:13.518626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.493122 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"simulator-w4-2f3705-kserve-self-signed-certs\"" Nov 01 22:35:13.556073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.555992 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:13.561199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.561163 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2cxs\" (UniqueName: \"kubernetes.io/projected/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-kube-api-access-f2cxs\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.561335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.561223 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tmp-dir\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.561335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.561269 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-dshm\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.561335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.561327 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tls-certs\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.561446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.561369 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-home\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.561446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.561393 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-model-cache\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.675611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.674230 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tls-certs\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.675611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.674321 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-home\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.675611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.674365 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-model-cache\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.675611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.674410 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2cxs\" (UniqueName: \"kubernetes.io/projected/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-kube-api-access-f2cxs\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.675611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.674468 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tmp-dir\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.675611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.674531 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-dshm\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.675611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.675594 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-model-cache\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.676859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.676838 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tmp-dir\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.698704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.681878 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-dshm\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.698704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.682186 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-home\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.714694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.713956 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tls-certs\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.745011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.744349 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 22:35:13.766602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.761629 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 22:35:13.771817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.768277 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2cxs\" (UniqueName: \"kubernetes.io/projected/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-kube-api-access-f2cxs\") pod \"simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:13.862756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:13.862669 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:14.347520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.347442 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:14.612673 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:35:14.603782 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:35:14.634599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.633534 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8"] Nov 01 22:35:14.643764 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:14.643094 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbd6472b_add6_4a3c_8fff_d09e0b182ba1.slice/crio-ca0885f5440b13d5d304c52a2750376bc4dc016ba8525428bae865c5a16b0aff WatchSource:0}: Error finding container ca0885f5440b13d5d304c52a2750376bc4dc016ba8525428bae865c5a16b0aff: Status 404 returned error can't find the container with id ca0885f5440b13d5d304c52a2750376bc4dc016ba8525428bae865c5a16b0aff Nov 01 22:35:14.731942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.725155 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 22:35:14.740016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.739988 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 22:35:14.830598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.829849 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" event={"ID":"bbd6472b-add6-4a3c-8fff-d09e0b182ba1","Type":"ContainerStarted","Data":"30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca"} Nov 01 22:35:14.830598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.829888 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" event={"ID":"bbd6472b-add6-4a3c-8fff-d09e0b182ba1","Type":"ContainerStarted","Data":"ca0885f5440b13d5d304c52a2750376bc4dc016ba8525428bae865c5a16b0aff"} Nov 01 22:35:14.855966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.855900 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podStartSLOduration=1.855880493 podStartE2EDuration="1.855880493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:14.853540069 +0000 UTC m=+680.281418003" watchObservedRunningTime="2026-11-01 22:35:14.855880493 +0000 UTC m=+680.283758425" Nov 01 22:35:14.863354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:14.863029 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:15.112844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:15.112797 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:15.960750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:15.960674 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:16.146253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:16.146160 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:16.639799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:16.639761 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:16.809687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:16.809647 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:17.024880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.024844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:17.170619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.168058 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:17.598516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.598398 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:17.799160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.799079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:17.857511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.857432 4981 generic.go:356] "Generic (PLEG): container finished" podID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerID="815bba694e5437fd06c34f709cf97999d5fc90151a8d55762db0ef48f430e587" exitCode=0 Nov 01 22:35:17.857511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.857499 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" event={"ID":"1e14a2a1-e3bd-4bb4-a477-06d73f87df06","Type":"ContainerDied","Data":"815bba694e5437fd06c34f709cf97999d5fc90151a8d55762db0ef48f430e587"} Nov 01 22:35:17.857983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.857966 4981 scope.go:122] "RemoveContainer" containerID="815bba694e5437fd06c34f709cf97999d5fc90151a8d55762db0ef48f430e587" Nov 01 22:35:17.859745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.859711 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:17.859835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.859806 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:17.859873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.859855 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:17.936817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:17.936780 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:18.868020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:18.867978 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" event={"ID":"1e14a2a1-e3bd-4bb4-a477-06d73f87df06","Type":"ContainerStarted","Data":"8ada89906faa209141894878630924d9a00dc4465d8ecc2ac3c086da220ce83a"} Nov 01 22:35:18.932124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:18.932091 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:35:19.006662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:18.984394 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:19.112201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.112164 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z"] Nov 01 22:35:19.116957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.116933 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.127034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.126922 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"kube-root-ca.crt\"" Nov 01 22:35:19.127185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.127073 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"simulator-w5-e26733-kserve-self-signed-certs\"" Nov 01 22:35:19.127185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.127174 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"openshift-service-ca.crt\"" Nov 01 22:35:19.132407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.127759 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"default-dockercfg-k6rcx\"" Nov 01 22:35:19.132407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.131938 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z"] Nov 01 22:35:19.157229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.157172 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3da20-7cee-42f5-955f-0e90546fadde-tls-certs\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.157396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.157229 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-model-cache\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.157396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.157323 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-home\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.157396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.157390 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-tmp-dir\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.157516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.157438 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmnl\" (UniqueName: \"kubernetes.io/projected/6fa3da20-7cee-42f5-955f-0e90546fadde-kube-api-access-thmnl\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.157516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.157472 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-dshm\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.206130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.206088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:19.260124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260088 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3da20-7cee-42f5-955f-0e90546fadde-tls-certs\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.260302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260131 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-model-cache\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.260302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260203 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-home\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.260302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260253 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-tmp-dir\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.260302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260290 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmnl\" (UniqueName: \"kubernetes.io/projected/6fa3da20-7cee-42f5-955f-0e90546fadde-kube-api-access-thmnl\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.260511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260321 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-dshm\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.260650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260625 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-model-cache\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.260764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260737 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-home\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.261003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.260985 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-tmp-dir\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.263102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.263071 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-dshm\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.264207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.264182 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3da20-7cee-42f5-955f-0e90546fadde-tls-certs\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.270914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.270870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmnl\" (UniqueName: \"kubernetes.io/projected/6fa3da20-7cee-42f5-955f-0e90546fadde-kube-api-access-thmnl\") pod \"simulator-w5-e26733-kserve-f668d978b-br58z\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.442539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.442503 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:19.775605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.775509 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z"] Nov 01 22:35:19.879323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.878422 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" event={"ID":"6fa3da20-7cee-42f5-955f-0e90546fadde","Type":"ContainerStarted","Data":"a61d71a69d8bb4581bf90789c8a4771dc1a189deebe192f76389c972902899a8"} Nov 01 22:35:19.879323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.878478 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" event={"ID":"6fa3da20-7cee-42f5-955f-0e90546fadde","Type":"ContainerStarted","Data":"56540d0d675d5784de5416052e93fe29d09f68445c76b6253affa5fbc4de3958"} Nov 01 22:35:19.884485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.882676 4981 generic.go:356] "Generic (PLEG): container finished" podID="6182a7fa-5369-41de-9122-45ca7836627d" containerID="9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6" exitCode=0 Nov 01 22:35:19.884485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.883335 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" event={"ID":"6182a7fa-5369-41de-9122-45ca7836627d","Type":"ContainerDied","Data":"9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6"} Nov 01 22:35:19.884485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.883695 4981 scope.go:122] "RemoveContainer" containerID="9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6" Nov 01 22:35:19.888658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.888626 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:19.888860 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.888836 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:19.889021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.888997 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:19.931774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.931745 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:35:19.932276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.932242 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:35:19.932861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:19.932814 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podStartSLOduration=0.932796346 podStartE2EDuration="932.796346ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:19.928258194 +0000 UTC m=+685.356136125" watchObservedRunningTime="2026-11-01 22:35:19.932796346 +0000 UTC m=+685.360674278" Nov 01 22:35:20.136564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.136485 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:20.442988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.442946 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:20.932601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.932152 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:35:20.932601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.932205 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:35:20.932601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.932376 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" event={"ID":"6182a7fa-5369-41de-9122-45ca7836627d","Type":"ContainerStarted","Data":"f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435"} Nov 01 22:35:20.934596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.934558 4981 generic.go:356] "Generic (PLEG): container finished" podID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerID="d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a" exitCode=0 Nov 01 22:35:20.934698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.934658 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" event={"ID":"ad33ddb9-b1b9-4b1f-9691-607def69c690","Type":"ContainerDied","Data":"d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a"} Nov 01 22:35:20.935293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.935105 4981 scope.go:122] "RemoveContainer" containerID="d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a" Nov 01 22:35:20.937020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.936644 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:20.937020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.936711 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:20.937020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:20.936743 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:21.133751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:21.129477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:21.209978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:21.208862 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:35:21.223101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:21.212817 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.223101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:21.212873 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:35:21.932535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:21.932036 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:35:21.932535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:21.932104 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:35:21.960873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:21.960840 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" event={"ID":"ad33ddb9-b1b9-4b1f-9691-607def69c690","Type":"ContainerStarted","Data":"d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967"} Nov 01 22:35:22.127608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.127046 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:22.212659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.210024 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.212659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.210088 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:35:22.338608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.338452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:22.704920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.704831 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:35:22.705508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.705204 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.705508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.705254 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 22:35:22.834509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.834477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:22.937404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.936951 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:35:22.937404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:22.937035 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:35:23.032076 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.031912 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:23.213250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.212681 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.213250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.212745 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:35:23.349230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.349144 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb"] Nov 01 22:35:23.355510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.355480 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:23.364879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.364849 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb"] Nov 01 22:35:23.365221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.365165 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.371686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.371648 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"kube-root-ca.crt\"" Nov 01 22:35:23.371925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.371704 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"default-dockercfg-s59jf\"" Nov 01 22:35:23.371925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.371714 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"simulator-w6-927b68-kserve-self-signed-certs\"" Nov 01 22:35:23.371925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.371826 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"openshift-service-ca.crt\"" Nov 01 22:35:23.430143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.429628 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-model-cache\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.430143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.429674 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td8dx\" (UniqueName: \"kubernetes.io/projected/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-kube-api-access-td8dx\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.430143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.429779 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-home\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.430143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.429819 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-dshm\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.430143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.429910 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tls-certs\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.430143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.429939 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tmp-dir\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.533120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.532246 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tls-certs\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.533120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.532302 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tmp-dir\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.533120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.532353 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-model-cache\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.533120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.532381 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td8dx\" (UniqueName: \"kubernetes.io/projected/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-kube-api-access-td8dx\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.533120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.532520 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-home\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.533120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.532566 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-dshm\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.533999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.533966 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tmp-dir\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.551508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.534718 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-model-cache\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.551725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.534949 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-home\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.640063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.639988 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tls-certs\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.667105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.667050 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-dshm\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.670092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.670069 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td8dx\" (UniqueName: \"kubernetes.io/projected/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-kube-api-access-td8dx\") pod \"simulator-w6-927b68-kserve-69787658df-mmvbb\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.699729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.699290 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.699729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.699344 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 22:35:23.745023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.744745 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:23.856600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.852098 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:23.864956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.863613 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:23.981600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.946767 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 22:35:23.981600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:23.946827 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 22:35:24.022300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.022006 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:24.138838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.138797 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:35:24.209335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.208872 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:35:24.209335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.208927 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:35:24.253333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.253291 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:24.351055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.351023 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb"] Nov 01 22:35:24.708688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.708244 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:35:24.708688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.708300 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 22:35:24.890653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.890253 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:24.955671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.955642 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:35:24.956992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.956974 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:35:24.957978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:24.957958 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 22:35:25.051602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.049853 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:25.154017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.153916 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" event={"ID":"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd","Type":"ContainerStarted","Data":"18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009"} Nov 01 22:35:25.154017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.153971 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" event={"ID":"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd","Type":"ContainerStarted","Data":"5c97a5597cf5e047dfecb1e3050db8b6a461565fd47df109df049183e9ae7a80"} Nov 01 22:35:25.191600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.191061 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podStartSLOduration=2.191041945 podStartE2EDuration="2.191041945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:25.190328502 +0000 UTC m=+690.618206435" watchObservedRunningTime="2026-11-01 22:35:25.191041945 +0000 UTC m=+690.618919878" Nov 01 22:35:25.215600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.211564 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 22:35:25.215600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.211650 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 22:35:25.620600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.619680 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:35:25.699600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.699008 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:35:25.699600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.699071 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 22:35:25.745551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:25.745516 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:26.018142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.018106 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:26.286653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.284398 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:26.352470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.352435 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:35:26.354490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.354469 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:35:26.622634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.622554 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:35:26.698396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.697942 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 22:35:26.698396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.698012 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 22:35:26.801614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.799795 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:45256->10.134.0.44:15021: read: connection reset by peer" start-of-body= Nov 01 22:35:26.801614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.799857 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:45256->10.134.0.44:15021: read: connection reset by peer" Nov 01 22:35:26.837521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.837494 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:26.987600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:26.983036 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:27.100661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.100626 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:27.167935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.167905 4981 generic.go:356] "Generic (PLEG): container finished" podID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerID="7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008" exitCode=0 Nov 01 22:35:27.168126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.167980 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" event={"ID":"3c13c7e4-17a5-470b-becd-8c18896915d3","Type":"ContainerDied","Data":"7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008"} Nov 01 22:35:27.168564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.168546 4981 scope.go:122] "RemoveContainer" containerID="7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008" Nov 01 22:35:27.169853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.169824 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:27.169959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.169887 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:27.169959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.169926 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:35:27.704438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.704404 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:35:27.705569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.705546 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 22:35:27.863168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:27.863133 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:28.013692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.013597 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7"] Nov 01 22:35:28.017665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.017643 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.027329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.027305 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"kube-root-ca.crt\"" Nov 01 22:35:28.027476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.027351 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"openshift-service-ca.crt\"" Nov 01 22:35:28.027476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.027396 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"simulator-w1-f338e4-kserve-self-signed-certs\"" Nov 01 22:35:28.027592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.027499 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f338e4\"/\"default-dockercfg-td692\"" Nov 01 22:35:28.032484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.032462 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7"] Nov 01 22:35:28.073457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.073431 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:28.152309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.152266 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vqk\" (UniqueName: \"kubernetes.io/projected/a2e8cc95-9482-47a6-b8c2-2507116356d4-kube-api-access-z5vqk\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.152489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.152322 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-dshm\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.152489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.152463 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e8cc95-9482-47a6-b8c2-2507116356d4-tls-certs\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.152609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.152534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-tmp-dir\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.152658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.152612 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-home\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.152658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.152645 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-model-cache\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.225774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.224760 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" event={"ID":"3c13c7e4-17a5-470b-becd-8c18896915d3","Type":"ContainerStarted","Data":"6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162"} Nov 01 22:35:28.253726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.253690 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-tmp-dir\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.253911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.253753 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-home\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.253911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.253780 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-model-cache\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.253911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.253870 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vqk\" (UniqueName: \"kubernetes.io/projected/a2e8cc95-9482-47a6-b8c2-2507116356d4-kube-api-access-z5vqk\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.253911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.253901 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-dshm\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.254119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.253946 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e8cc95-9482-47a6-b8c2-2507116356d4-tls-certs\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.254119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.254097 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-tmp-dir\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.254244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.254157 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-home\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.254325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.254305 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-model-cache\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.256250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.256224 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-dshm\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.257689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.257663 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e8cc95-9482-47a6-b8c2-2507116356d4-tls-certs\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.289647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.287318 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vqk\" (UniqueName: \"kubernetes.io/projected/a2e8cc95-9482-47a6-b8c2-2507116356d4-kube-api-access-z5vqk\") pod \"simulator-w1-f338e4-kserve-795c68bcb7-2p9q7\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.331334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.331300 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:28.648455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.648428 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7"] Nov 01 22:35:28.649750 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:28.649725 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2e8cc95_9482_47a6_b8c2_2507116356d4.slice/crio-e8750972590f58fd72d4b24d9a698283b369da29af7352211734e45a51b5f65b WatchSource:0}: Error finding container e8750972590f58fd72d4b24d9a698283b369da29af7352211734e45a51b5f65b: Status 404 returned error can't find the container with id e8750972590f58fd72d4b24d9a698283b369da29af7352211734e45a51b5f65b Nov 01 22:35:28.935727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.935695 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:35:28.936244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.935910 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:35:28.936244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.935953 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 22:35:28.990601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:28.987138 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:29.240653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.239913 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" event={"ID":"a2e8cc95-9482-47a6-b8c2-2507116356d4","Type":"ContainerStarted","Data":"836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b"} Nov 01 22:35:29.240653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.239955 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" event={"ID":"a2e8cc95-9482-47a6-b8c2-2507116356d4","Type":"ContainerStarted","Data":"e8750972590f58fd72d4b24d9a698283b369da29af7352211734e45a51b5f65b"} Nov 01 22:35:29.262840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.262508 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podStartSLOduration=2.262490821 podStartE2EDuration="2.262490821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:29.261718259 +0000 UTC m=+694.689596193" watchObservedRunningTime="2026-11-01 22:35:29.262490821 +0000 UTC m=+694.690368753" Nov 01 22:35:29.331730 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.331696 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:29.443238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.443201 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:29.456297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.456271 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:29.899943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.899911 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:29.935969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.935935 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:35:29.936405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:29.936010 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 22:35:30.256991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:30.256963 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:35:30.699262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:30.699231 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:30.934908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:30.934872 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:35:30.935087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:30.934921 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 22:35:31.398600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:31.398478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:31.935369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:31.935332 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:35:31.935558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:31.935383 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 22:35:32.096092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:32.096056 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:32.934925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:32.934891 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 22:35:32.935394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:32.934963 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 22:35:32.971246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:32.971213 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:33.745640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:33.745608 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:33.757522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:33.757492 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:33.778731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:33.778668 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:33.938930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:33.938902 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:35:33.939308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:33.939263 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:35:33.939976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:33.939956 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:35:34.279078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:34.279053 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:35:34.577217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:34.577105 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:35.386950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:35.386917 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:35.485277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:35.485247 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:36.364236 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:36.364206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:37.219439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:37.219410 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:37.872048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:37.872008 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:38.331769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:38.331737 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:38.343853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:38.343826 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:38.563435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:38.563403 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:38.694600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:38.694532 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:39.310191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:39.310158 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:35:39.503459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:39.503432 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:40.392757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:40.392722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:41.151418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:41.151380 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:41.877762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:41.877728 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:42.541752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:42.541721 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:43.285024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:43.284989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:43.911623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:43.911560 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:44.525335 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:35:44.525300 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:35:44.578451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:44.578415 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:45.543851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.543815 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc"] Nov 01 22:35:45.548067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.548046 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.556546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.556181 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"premium-w5-e26733-kserve-self-signed-certs\"" Nov 01 22:35:45.563419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.563387 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-dshm\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.563538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.563453 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cbae16-acbd-43b9-a72c-008458724815-tls-certs\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.563620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.563561 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skhq\" (UniqueName: \"kubernetes.io/projected/32cbae16-acbd-43b9-a72c-008458724815-kube-api-access-2skhq\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.563715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.563698 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-home\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.563775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.563737 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-tmp-dir\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.563891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.563869 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-model-cache\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.564962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.564937 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc"] Nov 01 22:35:45.596158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.596132 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:45.665233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665201 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cbae16-acbd-43b9-a72c-008458724815-tls-certs\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.665233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665243 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skhq\" (UniqueName: \"kubernetes.io/projected/32cbae16-acbd-43b9-a72c-008458724815-kube-api-access-2skhq\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.665833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665290 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-home\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.665833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665311 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-tmp-dir\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.665833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665358 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-model-cache\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.665833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665420 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-dshm\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.665833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665777 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-home\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.665833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665823 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-tmp-dir\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.666077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.665861 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-model-cache\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.667663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.667642 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-dshm\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.667830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.667814 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cbae16-acbd-43b9-a72c-008458724815-tls-certs\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.673259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.673241 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skhq\" (UniqueName: \"kubernetes.io/projected/32cbae16-acbd-43b9-a72c-008458724815-kube-api-access-2skhq\") pod \"premium-w5-e26733-kserve-5b4c56bd75-694cc\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:45.723900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.723849 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:45.861866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:45.861788 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:46.146983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:46.146953 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc"] Nov 01 22:35:46.149970 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:46.149942 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32cbae16_acbd_43b9_a72c_008458724815.slice/crio-c2f55f56aa3b9850322636804aaeebfd78763ab052c3a911423f496b2d65fd0d WatchSource:0}: Error finding container c2f55f56aa3b9850322636804aaeebfd78763ab052c3a911423f496b2d65fd0d: Status 404 returned error can't find the container with id c2f55f56aa3b9850322636804aaeebfd78763ab052c3a911423f496b2d65fd0d Nov 01 22:35:46.338348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:46.338310 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" event={"ID":"32cbae16-acbd-43b9-a72c-008458724815","Type":"ContainerStarted","Data":"6871eb6391bd584232989b9ded063e04fae5b807df1e2f180155fe55e88c61e6"} Nov 01 22:35:46.338348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:46.338350 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" event={"ID":"32cbae16-acbd-43b9-a72c-008458724815","Type":"ContainerStarted","Data":"c2f55f56aa3b9850322636804aaeebfd78763ab052c3a911423f496b2d65fd0d"} Nov 01 22:35:46.356550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:46.356500 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podStartSLOduration=1.356487922 podStartE2EDuration="1.356487922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:46.354743145 +0000 UTC m=+711.782621076" watchObservedRunningTime="2026-11-01 22:35:46.356487922 +0000 UTC m=+711.784365851" Nov 01 22:35:46.376043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:46.375986 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:46.862130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:46.862095 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:47.176467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:47.176441 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:47.880847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:47.880817 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:48.367637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:48.367602 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:48.619473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:48.619400 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:49.469658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:49.469629 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:50.380675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:50.380644 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:51.074835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:51.074806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:51.573430 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:51.573399 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:51.888016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:51.887947 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:52.671736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:52.671699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:53.344242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:53.344214 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:54.046933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:54.046898 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:54.766033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:54.765991 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:55.307381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.307351 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx"] Nov 01 22:35:55.312392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.312373 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.319876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.319839 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"default-dockercfg-pb82b\"" Nov 01 22:35:55.320010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.319983 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"premium-w3-6ac182-kserve-self-signed-certs\"" Nov 01 22:35:55.320086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.320072 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"openshift-service-ca.crt\"" Nov 01 22:35:55.320143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.320092 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"kube-root-ca.crt\"" Nov 01 22:35:55.323104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.323083 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx"] Nov 01 22:35:55.350316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.350278 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:55.370952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.370929 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-model-cache\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.370952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.370957 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-tmp-dir\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.371090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.371075 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-dshm\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.371157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.371142 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae6a304-8020-419b-95ef-5d5242a94e0c-tls-certs\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.371197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.371177 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pb48\" (UniqueName: \"kubernetes.io/projected/dae6a304-8020-419b-95ef-5d5242a94e0c-kube-api-access-5pb48\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.371287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.371276 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-home\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472139 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-model-cache\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472175 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-tmp-dir\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472237 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-dshm\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472297 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae6a304-8020-419b-95ef-5d5242a94e0c-tls-certs\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472317 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pb48\" (UniqueName: \"kubernetes.io/projected/dae6a304-8020-419b-95ef-5d5242a94e0c-kube-api-access-5pb48\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472376 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-home\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472637 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-model-cache\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472652 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-tmp-dir\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.472859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.472834 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-home\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.474507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.474481 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-dshm\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.474867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.474847 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae6a304-8020-419b-95ef-5d5242a94e0c-tls-certs\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.480635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.480612 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pb48\" (UniqueName: \"kubernetes.io/projected/dae6a304-8020-419b-95ef-5d5242a94e0c-kube-api-access-5pb48\") pod \"premium-w3-6ac182-kserve-576db9bdb-vg5hx\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.626081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.625995 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:55.669303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.669249 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:55.862000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.861966 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:55.874264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.874242 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:55.904131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:55.904098 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx"] Nov 01 22:35:55.905562 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:35:55.905540 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddae6a304_8020_419b_95ef_5d5242a94e0c.slice/crio-a80a88afe266bb37afde8377d9988e2bc2d8d06d7c1e0db3af23b45d4e38c97d WatchSource:0}: Error finding container a80a88afe266bb37afde8377d9988e2bc2d8d06d7c1e0db3af23b45d4e38c97d: Status 404 returned error can't find the container with id a80a88afe266bb37afde8377d9988e2bc2d8d06d7c1e0db3af23b45d4e38c97d Nov 01 22:35:56.394172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:56.394137 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:56.394896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:56.394871 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" event={"ID":"dae6a304-8020-419b-95ef-5d5242a94e0c","Type":"ContainerStarted","Data":"f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02"} Nov 01 22:35:56.394989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:56.394913 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" event={"ID":"dae6a304-8020-419b-95ef-5d5242a94e0c","Type":"ContainerStarted","Data":"a80a88afe266bb37afde8377d9988e2bc2d8d06d7c1e0db3af23b45d4e38c97d"} Nov 01 22:35:56.406615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:56.406593 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:35:56.417082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:56.417048 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podStartSLOduration=1.4170366620000001 podStartE2EDuration="1.417036662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:56.415518806 +0000 UTC m=+721.843396736" watchObservedRunningTime="2026-11-01 22:35:56.417036662 +0000 UTC m=+721.844914592" Nov 01 22:35:56.594939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:56.594911 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:56.626955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:56.626927 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:35:57.594392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:57.594359 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:58.280283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:58.280257 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:59.070306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:59.070267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:35:59.839263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:35:59.839236 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:00.576397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:00.576366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:01.272128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:01.272085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:01.954269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:01.954237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:02.453381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.453344 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w"] Nov 01 22:36:02.459854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.459827 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.470297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.470272 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"default-dockercfg-zngpk\"" Nov 01 22:36:02.470456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.470325 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"openshift-service-ca.crt\"" Nov 01 22:36:02.470456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.470444 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"premium-w0-e5f47e-kserve-self-signed-certs\"" Nov 01 22:36:02.470610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.470416 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"kube-root-ca.crt\"" Nov 01 22:36:02.470657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.470622 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w"] Nov 01 22:36:02.508793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.508759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:02.555127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.555086 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-dshm\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.555297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.555212 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz76w\" (UniqueName: \"kubernetes.io/projected/1794effc-dda4-4f3c-a529-5fa2fc655d6b-kube-api-access-kz76w\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.555297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.555276 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tmp-dir\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.555369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.555310 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-model-cache\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.555410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.555390 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-home\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.555448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.555435 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tls-certs\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656047 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-dshm\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656159 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz76w\" (UniqueName: \"kubernetes.io/projected/1794effc-dda4-4f3c-a529-5fa2fc655d6b-kube-api-access-kz76w\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656190 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tmp-dir\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656233 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-model-cache\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656282 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-home\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656323 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tls-certs\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656610 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tmp-dir\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656727 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-home\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.656875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.656859 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-model-cache\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.658544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.658512 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-dshm\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.658810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.658792 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tls-certs\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.664841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.664818 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz76w\" (UniqueName: \"kubernetes.io/projected/1794effc-dda4-4f3c-a529-5fa2fc655d6b-kube-api-access-kz76w\") pod \"premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.774565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.774477 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:02.849421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:02.849390 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:03.052323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:03.052288 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w"] Nov 01 22:36:03.053962 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:03.053926 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1794effc_dda4_4f3c_a529_5fa2fc655d6b.slice/crio-167aaee2ed54c27ac576c2441c10a2bb1747052c462de931e750289d7e20e6b1 WatchSource:0}: Error finding container 167aaee2ed54c27ac576c2441c10a2bb1747052c462de931e750289d7e20e6b1: Status 404 returned error can't find the container with id 167aaee2ed54c27ac576c2441c10a2bb1747052c462de931e750289d7e20e6b1 Nov 01 22:36:03.434422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:03.434388 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" event={"ID":"1794effc-dda4-4f3c-a529-5fa2fc655d6b","Type":"ContainerStarted","Data":"33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269"} Nov 01 22:36:03.434726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:03.434435 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" event={"ID":"1794effc-dda4-4f3c-a529-5fa2fc655d6b","Type":"ContainerStarted","Data":"167aaee2ed54c27ac576c2441c10a2bb1747052c462de931e750289d7e20e6b1"} Nov 01 22:36:03.459834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:03.459785 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podStartSLOduration=1.459772005 podStartE2EDuration="1.459772005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:03.458868668 +0000 UTC m=+728.886746599" watchObservedRunningTime="2026-11-01 22:36:03.459772005 +0000 UTC m=+728.887649935" Nov 01 22:36:03.682762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:03.682732 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:03.775150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:03.775061 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:04.561771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:04.561729 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:05.433664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:05.433633 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:05.627133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:05.627102 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:36:05.639335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:05.639311 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:36:06.192770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:06.192740 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:06.461744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:06.461667 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:36:06.880496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:06.880419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:07.725646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:07.725609 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:08.922942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:08.922907 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:09.764383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:09.764342 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:10.471001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:10.470964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:11.259503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:11.259474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:12.267827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:12.267797 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:12.372536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:12.372499 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:12.775641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:12.775605 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:12.787960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:12.787928 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:13.234080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:13.234048 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:13.506529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:13.506445 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:36:13.982780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:13.982749 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:14.522830 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:36:14.522797 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:36:14.777162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:14.777072 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:15.461514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:15.461481 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:16.067372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:16.067335 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:16.705917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:16.705887 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:17.383644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:17.383603 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:18.044552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:18.044516 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:18.800477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:18.800436 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:19.465763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:19.465726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:20.174317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:20.174287 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:21.096501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:21.096473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:22.370066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:22.370021 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:23.195256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:23.195225 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:24.185234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:24.185195 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:25.657429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:25.657388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:26.572266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:26.572228 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:27.485958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:27.485918 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:28.465395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:28.465308 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:29.258328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:29.258300 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:30.028586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:30.028553 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:30.794437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:30.794399 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:31.568035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:31.567999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:32.263416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:32.263385 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:33.487494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:33.487460 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:34.300103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:34.300063 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:34.968246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:34.968213 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:35.699296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.699258 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc"] Nov 01 22:36:35.704495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.704473 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.710061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.710036 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:36:35.710192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.710081 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:36:35.710192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.710150 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:35.710192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.710159 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:36:35.711665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.711633 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc"] Nov 01 22:36:35.765467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.765437 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:35.814243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.814193 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29k6b\" (UniqueName: \"kubernetes.io/projected/b8d7ca94-df4d-4abb-b654-f883952fec10-kube-api-access-29k6b\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.814626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.814596 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.814736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.814642 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.915855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.915817 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29k6b\" (UniqueName: \"kubernetes.io/projected/b8d7ca94-df4d-4abb-b654-f883952fec10-kube-api-access-29k6b\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.916058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.915889 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.916058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.915910 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.916610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.916568 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.918276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.918255 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:35.924281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:35.924262 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29k6b\" (UniqueName: \"kubernetes.io/projected/b8d7ca94-df4d-4abb-b654-f883952fec10-kube-api-access-29k6b\") pod \"maas-api-revoke-sub-d9c4fd1c-9gzcc\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:36.023347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:36.023268 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:36.309388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:36.309364 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc"] Nov 01 22:36:36.311986 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:36.311958 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d7ca94_df4d_4abb_b654_f883952fec10.slice/crio-2573b9121a02a782c0d8a455029e9111e2541abcf9b47406160673768f67ae4a WatchSource:0}: Error finding container 2573b9121a02a782c0d8a455029e9111e2541abcf9b47406160673768f67ae4a: Status 404 returned error can't find the container with id 2573b9121a02a782c0d8a455029e9111e2541abcf9b47406160673768f67ae4a Nov 01 22:36:36.314006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:36.313991 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:36:36.577973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:36.577880 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:36.656517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:36.656479 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" event={"ID":"b8d7ca94-df4d-4abb-b654-f883952fec10","Type":"ContainerStarted","Data":"2573b9121a02a782c0d8a455029e9111e2541abcf9b47406160673768f67ae4a"} Nov 01 22:36:37.265659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:37.265628 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:38.063793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.063758 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:38.423010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.422974 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh"] Nov 01 22:36:38.431996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.431974 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.444748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.444728 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6ac182-openshift-default-dockercfg-mf2q8\"" Nov 01 22:36:38.468556 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.468523 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh"] Nov 01 22:36:38.549251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549212 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549260 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549403 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7731aca5-5577-4ec4-ba55-6c379ab1be21-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549554 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549601 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549700 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnpx\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-kube-api-access-hrnpx\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549744 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549857 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.549945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.549907 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651052 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651090 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651125 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnpx\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-kube-api-access-hrnpx\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651152 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651188 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651239 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651263 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651284 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651329 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7731aca5-5577-4ec4-ba55-6c379ab1be21-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651533 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651737 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651783 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.651943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.651919 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.652128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.652111 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7731aca5-5577-4ec4-ba55-6c379ab1be21-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.653752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.653729 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.654093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.654072 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.665349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.665325 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.665802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.665778 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnpx\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-kube-api-access-hrnpx\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.747738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.747655 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:38.934558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:38.934527 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:39.039460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.039426 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh"] Nov 01 22:36:39.040528 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:39.040501 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7731aca5_5577_4ec4_ba55_6c379ab1be21.slice/crio-44265ad8a37e3a015c6c203d0bacef4b862181c84f7454c6212406843ae49a1d WatchSource:0}: Error finding container 44265ad8a37e3a015c6c203d0bacef4b862181c84f7454c6212406843ae49a1d: Status 404 returned error can't find the container with id 44265ad8a37e3a015c6c203d0bacef4b862181c84f7454c6212406843ae49a1d Nov 01 22:36:39.042946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.042914 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:36:39.043031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.042976 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:36:39.043031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.043005 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:36:39.641653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.641555 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:39.714711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.714663 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" event={"ID":"7731aca5-5577-4ec4-ba55-6c379ab1be21","Type":"ContainerStarted","Data":"253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5"} Nov 01 22:36:39.715029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.714980 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" event={"ID":"7731aca5-5577-4ec4-ba55-6c379ab1be21","Type":"ContainerStarted","Data":"44265ad8a37e3a015c6c203d0bacef4b862181c84f7454c6212406843ae49a1d"} Nov 01 22:36:39.719658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.719597 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" event={"ID":"b8d7ca94-df4d-4abb-b654-f883952fec10","Type":"ContainerStarted","Data":"11eacb34a5fa9be2f9c3927755859478f00faf7de16bc8ad71f2848660f5d3dd"} Nov 01 22:36:39.739798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.739753 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podStartSLOduration=1.7397416799999998 podStartE2EDuration="1.73974168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:39.738669146 +0000 UTC m=+765.166547078" watchObservedRunningTime="2026-11-01 22:36:39.73974168 +0000 UTC m=+765.167619610" Nov 01 22:36:39.748325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.748305 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:39.755585 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:39.755534 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" podStartSLOduration=2.098555296 podStartE2EDuration="4.755522681s" totalImagesPullingTime="2.656967385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:39.754345789 +0000 UTC m=+765.182223719" watchObservedRunningTime="2026-11-01 22:36:39.755522681 +0000 UTC m=+765.183400662" Nov 01 22:36:40.398222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:40.398129 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:40.726997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:40.726960 4981 generic.go:356] "Generic (PLEG): container finished" podID="b8d7ca94-df4d-4abb-b654-f883952fec10" containerID="11eacb34a5fa9be2f9c3927755859478f00faf7de16bc8ad71f2848660f5d3dd" exitCode=0 Nov 01 22:36:40.733855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:40.733829 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" event={"ID":"b8d7ca94-df4d-4abb-b654-f883952fec10","Type":"ContainerDied","Data":"11eacb34a5fa9be2f9c3927755859478f00faf7de16bc8ad71f2848660f5d3dd"} Nov 01 22:36:40.749850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:40.749822 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:36:40.750000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:40.749869 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:36:41.167321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:41.167226 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:41.748967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:41.748925 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:36:41.749314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:41.749005 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:36:41.899492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:41.899450 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:41.930586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:41.930525 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:42.010658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.010557 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-xks-ca\") pod \"b8d7ca94-df4d-4abb-b654-f883952fec10\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " Nov 01 22:36:42.010840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.010788 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-openshift-ca\") pod \"b8d7ca94-df4d-4abb-b654-f883952fec10\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " Nov 01 22:36:42.010918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.010880 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d7ca94-df4d-4abb-b654-f883952fec10-kube-api-access-29k6b\" (UniqueName: \"kubernetes.io/projected/b8d7ca94-df4d-4abb-b654-f883952fec10-kube-api-access-29k6b\") pod \"b8d7ca94-df4d-4abb-b654-f883952fec10\" (UID: \"b8d7ca94-df4d-4abb-b654-f883952fec10\") " Nov 01 22:36:42.011214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.011185 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-openshift-ca" pod "b8d7ca94-df4d-4abb-b654-f883952fec10" (UID: "b8d7ca94-df4d-4abb-b654-f883952fec10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:42.011396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.011377 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:42.013287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.013241 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d7ca94-df4d-4abb-b654-f883952fec10-kube-api-access-29k6b" pod "b8d7ca94-df4d-4abb-b654-f883952fec10" (UID: "b8d7ca94-df4d-4abb-b654-f883952fec10"). InnerVolumeSpecName "kube-api-access-29k6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:42.013567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.013546 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-xks-ca" pod "b8d7ca94-df4d-4abb-b654-f883952fec10" (UID: "b8d7ca94-df4d-4abb-b654-f883952fec10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:42.112405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.112371 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29k6b\" (UniqueName: \"kubernetes.io/projected/b8d7ca94-df4d-4abb-b654-f883952fec10-kube-api-access-29k6b\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:42.112405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.112400 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d7ca94-df4d-4abb-b654-f883952fec10-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:42.594794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.594754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:42.740476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.740442 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" event={"ID":"b8d7ca94-df4d-4abb-b654-f883952fec10","Type":"ContainerDied","Data":"2573b9121a02a782c0d8a455029e9111e2541abcf9b47406160673768f67ae4a"} Nov 01 22:36:42.740665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.740489 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2573b9121a02a782c0d8a455029e9111e2541abcf9b47406160673768f67ae4a" Nov 01 22:36:42.740665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.740461 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc" Nov 01 22:36:42.748281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.748258 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:36:42.748410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:42.748305 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:36:43.346314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:43.346277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:43.748999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:43.748964 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:36:43.749225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:43.749019 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:36:44.069195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:44.069110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:44.527099 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:36:44.527061 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:36:44.748397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:44.748366 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 01 22:36:44.748608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:44.748423 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 01 22:36:44.873007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:44.872929 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:45.651587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:45.651537 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:45.752970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:45.752940 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:45.753384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:45.753362 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:45.754093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:45.754078 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 22:36:46.587491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:46.587463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:47.662123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:47.662088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:48.473794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:48.473762 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:49.297009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:49.296975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:50.182821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.182764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:50.820791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.820760 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch"] Nov 01 22:36:50.821317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.821303 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b8d7ca94-df4d-4abb-b654-f883952fec10" containerName="revoke-keys" Nov 01 22:36:50.821361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.821318 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d7ca94-df4d-4abb-b654-f883952fec10" containerName="revoke-keys" Nov 01 22:36:50.821415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.821407 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d7ca94-df4d-4abb-b654-f883952fec10" containerName="revoke-keys" Nov 01 22:36:50.824927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.824908 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:50.834002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.833978 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:36:50.834127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.834036 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:36:50.834385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.834357 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:36:50.834566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.834543 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:36:50.836607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.836557 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch"] Nov 01 22:36:50.923739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.923707 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztgr\" (UniqueName: \"kubernetes.io/projected/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-kube-api-access-nztgr\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:50.923900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.923757 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:50.923970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:50.923929 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.024954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.024924 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztgr\" (UniqueName: \"kubernetes.io/projected/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-kube-api-access-nztgr\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.025114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.024971 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.025195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.025161 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.025539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.025521 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.027445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.027428 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.046682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.046660 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztgr\" (UniqueName: \"kubernetes.io/projected/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-kube-api-access-nztgr\") pod \"maas-api-revoke-sub-cf67a2df-8qlch\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.142142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.142059 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:51.274520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.274489 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:51.419652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.419622 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch"] Nov 01 22:36:51.421035 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:51.421011 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cff08d_8fcb_4dd1_bbe9_9077d66ca8ab.slice/crio-71dfed4158441370d24f0449c6e054a932c24329f74beb4ba0bdea89eb75f5b0 WatchSource:0}: Error finding container 71dfed4158441370d24f0449c6e054a932c24329f74beb4ba0bdea89eb75f5b0: Status 404 returned error can't find the container with id 71dfed4158441370d24f0449c6e054a932c24329f74beb4ba0bdea89eb75f5b0 Nov 01 22:36:51.794500 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:51.794473 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" event={"ID":"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab","Type":"ContainerStarted","Data":"71dfed4158441370d24f0449c6e054a932c24329f74beb4ba0bdea89eb75f5b0"} Nov 01 22:36:52.185494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.185460 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l"] Nov 01 22:36:52.189453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.189435 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.197462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.197432 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l"] Nov 01 22:36:52.244229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.244195 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.244378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.244353 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-kube-api-access-v929c\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.244463 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.244437 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.312370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.312341 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:52.345974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.345940 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-kube-api-access-v929c\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.346110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.345989 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.346167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.346149 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.346636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.346613 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.348404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.348383 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.360180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.360140 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-kube-api-access-v929c\") pod \"maas-api-revoke-sub-4c9b0f2d-jbt7l\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.508185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.508109 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:52.786967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.786941 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l"] Nov 01 22:36:52.788194 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:52.788164 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5cf966_5553_4fe5_bee3_e4a0c8c5d7cf.slice/crio-05dc8e1bc5130aff92dbd38976e24e372cd34908ff684cdf3531d69e3cde92a3 WatchSource:0}: Error finding container 05dc8e1bc5130aff92dbd38976e24e372cd34908ff684cdf3531d69e3cde92a3: Status 404 returned error can't find the container with id 05dc8e1bc5130aff92dbd38976e24e372cd34908ff684cdf3531d69e3cde92a3 Nov 01 22:36:52.802062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.802034 4981 generic.go:356] "Generic (PLEG): container finished" podID="73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" containerID="56c4d500ea0817b8364e7f3b151135fdfa4bad0805f9c88881de54f97bd9aab3" exitCode=0 Nov 01 22:36:52.802165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.802119 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" event={"ID":"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab","Type":"ContainerDied","Data":"56c4d500ea0817b8364e7f3b151135fdfa4bad0805f9c88881de54f97bd9aab3"} Nov 01 22:36:52.803469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.803447 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" event={"ID":"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf","Type":"ContainerStarted","Data":"05dc8e1bc5130aff92dbd38976e24e372cd34908ff684cdf3531d69e3cde92a3"} Nov 01 22:36:52.977731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:52.977697 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:53.230769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.230739 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj"] Nov 01 22:36:53.234935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.234908 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.244459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.244434 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj"] Nov 01 22:36:53.357057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.356979 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.357057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.357022 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmm2\" (UniqueName: \"kubernetes.io/projected/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-kube-api-access-hwmm2\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.357230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.357161 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.458530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.458494 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.458726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.458659 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.458726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.458696 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmm2\" (UniqueName: \"kubernetes.io/projected/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-kube-api-access-hwmm2\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.459353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.459328 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.461039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.461020 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.467023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.467001 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmm2\" (UniqueName: \"kubernetes.io/projected/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-kube-api-access-hwmm2\") pod \"maas-api-revoke-sub-155961b8-gsblj\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.551907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.551878 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:53.760135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.760085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:53.811321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.811289 4981 generic.go:356] "Generic (PLEG): container finished" podID="0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" containerID="8ab69facae30df7528e8915647d81dd85515f7f3e06633dbaeb090691ef5c2ca" exitCode=0 Nov 01 22:36:53.812362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.811940 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" event={"ID":"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf","Type":"ContainerDied","Data":"8ab69facae30df7528e8915647d81dd85515f7f3e06633dbaeb090691ef5c2ca"} Nov 01 22:36:53.846020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.845989 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj"] Nov 01 22:36:53.849347 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:53.849317 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b8ae50_f70e_464b_95fc_d68bc5f8d7d7.slice/crio-93647f1ad67740dfbde34a8138eca8dd420447c9841dd68f3daee51af23184fe WatchSource:0}: Error finding container 93647f1ad67740dfbde34a8138eca8dd420447c9841dd68f3daee51af23184fe: Status 404 returned error can't find the container with id 93647f1ad67740dfbde34a8138eca8dd420447c9841dd68f3daee51af23184fe Nov 01 22:36:53.907053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.907022 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf"] Nov 01 22:36:53.923087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.923056 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf"] Nov 01 22:36:53.923246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.923197 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:53.961017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.960995 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:53.964158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.964135 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:53.964272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.964198 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ccpm\" (UniqueName: \"kubernetes.io/projected/fb0ad2d7-1848-479c-9ece-29970c872a7e-kube-api-access-4ccpm\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:53.964342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:53.964328 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.065171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065142 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-openshift-ca\") pod \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " Nov 01 22:36:54.065310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065283 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-kube-api-access-nztgr\" (UniqueName: \"kubernetes.io/projected/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-kube-api-access-nztgr\") pod \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " Nov 01 22:36:54.065346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065310 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-xks-ca\") pod \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\" (UID: \"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab\") " Nov 01 22:36:54.065556 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065539 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.065621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065565 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-openshift-ca" pod "73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" (UID: "73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:54.065694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065677 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ccpm\" (UniqueName: \"kubernetes.io/projected/fb0ad2d7-1848-479c-9ece-29970c872a7e-kube-api-access-4ccpm\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.065842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065825 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.065924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.065909 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:54.066293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.066270 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.067486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.067465 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-xks-ca" pod "73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" (UID: "73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:54.067592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.067503 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-kube-api-access-nztgr" pod "73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" (UID: "73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab"). InnerVolumeSpecName "kube-api-access-nztgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:54.068051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.068032 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.073239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.073217 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ccpm\" (UniqueName: \"kubernetes.io/projected/fb0ad2d7-1848-479c-9ece-29970c872a7e-kube-api-access-4ccpm\") pod \"maas-api-revoke-sub-59f9c635-fr4nf\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.167044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.166979 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztgr\" (UniqueName: \"kubernetes.io/projected/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-kube-api-access-nztgr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:54.167044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.167005 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:54.241294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.241259 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:54.520364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.520326 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf"] Nov 01 22:36:54.520748 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:54.520721 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0ad2d7_1848_479c_9ece_29970c872a7e.slice/crio-4ab32d3c70a05e6cbfaee6c3469468ca09b4f281b7fef8950b1165ca2d9f73a9 WatchSource:0}: Error finding container 4ab32d3c70a05e6cbfaee6c3469468ca09b4f281b7fef8950b1165ca2d9f73a9: Status 404 returned error can't find the container with id 4ab32d3c70a05e6cbfaee6c3469468ca09b4f281b7fef8950b1165ca2d9f73a9 Nov 01 22:36:54.568553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.568521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:54.587862 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.587839 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4"] Nov 01 22:36:54.588401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.588389 4981 cpu_manager.go:414] "containerMap: removing container" podUID="73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" containerName="revoke-keys" Nov 01 22:36:54.588443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.588403 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" containerName="revoke-keys" Nov 01 22:36:54.588514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.588505 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" containerName="revoke-keys" Nov 01 22:36:54.593182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.593167 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.600452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.600424 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4"] Nov 01 22:36:54.672886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.672855 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.672886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.672888 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jxmx\" (UniqueName: \"kubernetes.io/projected/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-kube-api-access-7jxmx\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.673086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.672972 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.774268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.773992 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.774268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.774037 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jxmx\" (UniqueName: \"kubernetes.io/projected/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-kube-api-access-7jxmx\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.774268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.774106 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.774729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.774709 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.776367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.776349 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.783228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.783209 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jxmx\" (UniqueName: \"kubernetes.io/projected/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-kube-api-access-7jxmx\") pod \"maas-api-revoke-sub-0cfa5fa7-npmx4\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.819697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.819668 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" Nov 01 22:36:54.819850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.819704 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch" event={"ID":"73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab","Type":"ContainerDied","Data":"71dfed4158441370d24f0449c6e054a932c24329f74beb4ba0bdea89eb75f5b0"} Nov 01 22:36:54.819850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.819754 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71dfed4158441370d24f0449c6e054a932c24329f74beb4ba0bdea89eb75f5b0" Nov 01 22:36:54.821449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.821418 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" event={"ID":"fb0ad2d7-1848-479c-9ece-29970c872a7e","Type":"ContainerStarted","Data":"581e1ae6ac390811548547fb3641c0f2be9969f301f1152188a47622db34c311"} Nov 01 22:36:54.821561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.821485 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" event={"ID":"fb0ad2d7-1848-479c-9ece-29970c872a7e","Type":"ContainerStarted","Data":"4ab32d3c70a05e6cbfaee6c3469468ca09b4f281b7fef8950b1165ca2d9f73a9"} Nov 01 22:36:54.823506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.823486 4981 generic.go:356] "Generic (PLEG): container finished" podID="c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" containerID="5dcb11bd76fadb9e92a44af60669c7fa01e5a93ae6bfa17cb68ab120a00ebc73" exitCode=0 Nov 01 22:36:54.823643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.823589 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" event={"ID":"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7","Type":"ContainerDied","Data":"5dcb11bd76fadb9e92a44af60669c7fa01e5a93ae6bfa17cb68ab120a00ebc73"} Nov 01 22:36:54.823643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.823628 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" event={"ID":"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7","Type":"ContainerStarted","Data":"93647f1ad67740dfbde34a8138eca8dd420447c9841dd68f3daee51af23184fe"} Nov 01 22:36:54.845740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.845673 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" podStartSLOduration=1.845659306 podStartE2EDuration="1.845659306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:54.844202684 +0000 UTC m=+780.272080616" watchObservedRunningTime="2026-11-01 22:36:54.845659306 +0000 UTC m=+780.273537237" Nov 01 22:36:54.909390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.909358 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:54.964969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:54.964940 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:55.080956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.080920 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-openshift-ca\") pod \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " Nov 01 22:36:55.081095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.080977 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-xks-ca\") pod \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " Nov 01 22:36:55.081138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.081114 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-kube-api-access-v929c\") pod \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\" (UID: \"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf\") " Nov 01 22:36:55.081360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.081330 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-openshift-ca" pod "0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" (UID: "0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:55.081716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.081693 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:55.083158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.083130 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-xks-ca" pod "0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" (UID: "0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:55.083271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.083177 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-kube-api-access-v929c" pod "0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" (UID: "0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf"). InnerVolumeSpecName "kube-api-access-v929c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:55.182539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.182507 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v929c\" (UniqueName: \"kubernetes.io/projected/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-kube-api-access-v929c\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:55.182539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.182537 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:55.189067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.189041 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4"] Nov 01 22:36:55.190680 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:36:55.190652 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f8a78a_22ff_4d14_825d_e1d63f4d8a8e.slice/crio-6d057aa30dbafc31d0082025379dacb18d32577bab2dd1f59287509a8f68b22b WatchSource:0}: Error finding container 6d057aa30dbafc31d0082025379dacb18d32577bab2dd1f59287509a8f68b22b: Status 404 returned error can't find the container with id 6d057aa30dbafc31d0082025379dacb18d32577bab2dd1f59287509a8f68b22b Nov 01 22:36:55.279239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.279211 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:55.831794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.831756 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" event={"ID":"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e","Type":"ContainerStarted","Data":"c6136a0262f640860d237906f1e5dff993ef3610c7ed52c6be47795e2532e19c"} Nov 01 22:36:55.832068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.831817 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" event={"ID":"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e","Type":"ContainerStarted","Data":"6d057aa30dbafc31d0082025379dacb18d32577bab2dd1f59287509a8f68b22b"} Nov 01 22:36:55.833047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.833025 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" event={"ID":"0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf","Type":"ContainerDied","Data":"05dc8e1bc5130aff92dbd38976e24e372cd34908ff684cdf3531d69e3cde92a3"} Nov 01 22:36:55.833147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.833062 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05dc8e1bc5130aff92dbd38976e24e372cd34908ff684cdf3531d69e3cde92a3" Nov 01 22:36:55.833147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.833028 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l" Nov 01 22:36:55.834554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.834526 4981 generic.go:356] "Generic (PLEG): container finished" podID="fb0ad2d7-1848-479c-9ece-29970c872a7e" containerID="581e1ae6ac390811548547fb3641c0f2be9969f301f1152188a47622db34c311" exitCode=0 Nov 01 22:36:55.834681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.834558 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" event={"ID":"fb0ad2d7-1848-479c-9ece-29970c872a7e","Type":"ContainerDied","Data":"581e1ae6ac390811548547fb3641c0f2be9969f301f1152188a47622db34c311"} Nov 01 22:36:55.851915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.851870 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" podStartSLOduration=1.851855427 podStartE2EDuration="1.851855427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:55.849458093 +0000 UTC m=+781.277336024" watchObservedRunningTime="2026-11-01 22:36:55.851855427 +0000 UTC m=+781.279733357" Nov 01 22:36:55.957036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.957006 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:55.965256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:55.965215 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:56.093097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.093067 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-kube-api-access-hwmm2\" (UniqueName: \"kubernetes.io/projected/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-kube-api-access-hwmm2\") pod \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " Nov 01 22:36:56.093256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.093170 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-openshift-ca\") pod \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " Nov 01 22:36:56.093342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.093261 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-xks-ca\") pod \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\" (UID: \"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7\") " Nov 01 22:36:56.093528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.093505 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-openshift-ca" pod "c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" (UID: "c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:56.093754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.093728 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:56.095342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.095314 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-xks-ca" pod "c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" (UID: "c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:56.095423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.095353 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-kube-api-access-hwmm2" pod "c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" (UID: "c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7"). InnerVolumeSpecName "kube-api-access-hwmm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:56.194984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.194948 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:56.194984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.194980 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwmm2\" (UniqueName: \"kubernetes.io/projected/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7-kube-api-access-hwmm2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:56.708240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.708201 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:56.842390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.842357 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" Nov 01 22:36:56.842390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.842374 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj" event={"ID":"c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7","Type":"ContainerDied","Data":"93647f1ad67740dfbde34a8138eca8dd420447c9841dd68f3daee51af23184fe"} Nov 01 22:36:56.842639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.842429 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93647f1ad67740dfbde34a8138eca8dd420447c9841dd68f3daee51af23184fe" Nov 01 22:36:56.843970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.843942 4981 generic.go:356] "Generic (PLEG): container finished" podID="72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" containerID="c6136a0262f640860d237906f1e5dff993ef3610c7ed52c6be47795e2532e19c" exitCode=0 Nov 01 22:36:56.844091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:56.844027 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" event={"ID":"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e","Type":"ContainerDied","Data":"c6136a0262f640860d237906f1e5dff993ef3610c7ed52c6be47795e2532e19c"} Nov 01 22:36:57.018464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.018434 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:57.105615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.105543 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-xks-ca\") pod \"fb0ad2d7-1848-479c-9ece-29970c872a7e\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " Nov 01 22:36:57.105773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.105676 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0ad2d7-1848-479c-9ece-29970c872a7e-kube-api-access-4ccpm\" (UniqueName: \"kubernetes.io/projected/fb0ad2d7-1848-479c-9ece-29970c872a7e-kube-api-access-4ccpm\") pod \"fb0ad2d7-1848-479c-9ece-29970c872a7e\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " Nov 01 22:36:57.105773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.105760 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-openshift-ca\") pod \"fb0ad2d7-1848-479c-9ece-29970c872a7e\" (UID: \"fb0ad2d7-1848-479c-9ece-29970c872a7e\") " Nov 01 22:36:57.106180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.106138 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-openshift-ca" pod "fb0ad2d7-1848-479c-9ece-29970c872a7e" (UID: "fb0ad2d7-1848-479c-9ece-29970c872a7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:57.106325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.106300 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:57.107840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.107808 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-xks-ca" pod "fb0ad2d7-1848-479c-9ece-29970c872a7e" (UID: "fb0ad2d7-1848-479c-9ece-29970c872a7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:57.107840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.107827 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0ad2d7-1848-479c-9ece-29970c872a7e-kube-api-access-4ccpm" pod "fb0ad2d7-1848-479c-9ece-29970c872a7e" (UID: "fb0ad2d7-1848-479c-9ece-29970c872a7e"). InnerVolumeSpecName "kube-api-access-4ccpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:57.207994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.207961 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ccpm\" (UniqueName: \"kubernetes.io/projected/fb0ad2d7-1848-479c-9ece-29970c872a7e-kube-api-access-4ccpm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:57.207994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.207990 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb0ad2d7-1848-479c-9ece-29970c872a7e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:57.368692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.368613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:57.851560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.851531 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" Nov 01 22:36:57.851960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.851588 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf" event={"ID":"fb0ad2d7-1848-479c-9ece-29970c872a7e","Type":"ContainerDied","Data":"4ab32d3c70a05e6cbfaee6c3469468ca09b4f281b7fef8950b1165ca2d9f73a9"} Nov 01 22:36:57.851960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.851645 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ab32d3c70a05e6cbfaee6c3469468ca09b4f281b7fef8950b1165ca2d9f73a9" Nov 01 22:36:57.951104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:57.951074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:58.004207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.004179 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:58.030756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.030709 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:58.117972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.117884 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-openshift-ca\") pod \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " Nov 01 22:36:58.118142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.118028 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-xks-ca\") pod \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " Nov 01 22:36:58.118142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.118052 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-kube-api-access-7jxmx\" (UniqueName: \"kubernetes.io/projected/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-kube-api-access-7jxmx\") pod \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\" (UID: \"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e\") " Nov 01 22:36:58.118278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.118255 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-openshift-ca" pod "72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" (UID: "72f8a78a-22ff-4d14-825d-e1d63f4d8a8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:36:58.118445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.118427 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:58.120217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.120182 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-xks-ca" pod "72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" (UID: "72f8a78a-22ff-4d14-825d-e1d63f4d8a8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:36:58.120217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.120201 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-kube-api-access-7jxmx" pod "72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" (UID: "72f8a78a-22ff-4d14-825d-e1d63f4d8a8e"). InnerVolumeSpecName "kube-api-access-7jxmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:36:58.153945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.153905 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:58.219559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.219516 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:58.219559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.219559 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jxmx\" (UniqueName: \"kubernetes.io/projected/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e-kube-api-access-7jxmx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:36:58.858388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.858355 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" Nov 01 22:36:58.858388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.858379 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4" event={"ID":"72f8a78a-22ff-4d14-825d-e1d63f4d8a8e","Type":"ContainerDied","Data":"6d057aa30dbafc31d0082025379dacb18d32577bab2dd1f59287509a8f68b22b"} Nov 01 22:36:58.858955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.858422 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d057aa30dbafc31d0082025379dacb18d32577bab2dd1f59287509a8f68b22b" Nov 01 22:36:58.878788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:58.878760 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:36:59.570780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:36:59.570745 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:00.322286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:00.322233 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:01.157184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:01.157140 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:02.043286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.043238 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:02.517108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.517068 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85"] Nov 01 22:37:02.517946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.517916 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" containerName="revoke-keys" Nov 01 22:37:02.517998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.517950 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" containerName="revoke-keys" Nov 01 22:37:02.517998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.517970 4981 cpu_manager.go:414] "containerMap: removing container" podUID="72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" containerName="revoke-keys" Nov 01 22:37:02.517998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.517980 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" containerName="revoke-keys" Nov 01 22:37:02.518104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518021 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" containerName="revoke-keys" Nov 01 22:37:02.518104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518030 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" containerName="revoke-keys" Nov 01 22:37:02.518104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518047 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fb0ad2d7-1848-479c-9ece-29970c872a7e" containerName="revoke-keys" Nov 01 22:37:02.518104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518055 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0ad2d7-1848-479c-9ece-29970c872a7e" containerName="revoke-keys" Nov 01 22:37:02.518232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518205 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" containerName="revoke-keys" Nov 01 22:37:02.518232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518219 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" containerName="revoke-keys" Nov 01 22:37:02.518296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518234 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" containerName="revoke-keys" Nov 01 22:37:02.518296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.518243 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0ad2d7-1848-479c-9ece-29970c872a7e" containerName="revoke-keys" Nov 01 22:37:02.522684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.522661 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.528494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.528462 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:02.528678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.528569 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:37:02.529472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.529442 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85"] Nov 01 22:37:02.535069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.534951 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:02.535312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.535272 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:02.675942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.675905 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.676261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.676236 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpn7\" (UniqueName: \"kubernetes.io/projected/38d63c56-fb4c-4dcd-b000-59dd90860843-kube-api-access-tmpn7\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.676480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.676461 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.810651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.802164 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.810651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.802294 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.810651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.802390 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpn7\" (UniqueName: \"kubernetes.io/projected/38d63c56-fb4c-4dcd-b000-59dd90860843-kube-api-access-tmpn7\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.810651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.803526 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.839510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.839460 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.839711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.839533 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26"] Nov 01 22:37:02.869600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.858395 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpn7\" (UniqueName: \"kubernetes.io/projected/38d63c56-fb4c-4dcd-b000-59dd90860843-kube-api-access-tmpn7\") pod \"maas-api-revoke-sub-f3a46cba-9js85\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:02.869600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.868362 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26"] Nov 01 22:37:02.869600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:02.868373 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.055270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.055225 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcw5\" (UniqueName: \"kubernetes.io/projected/bccf430b-df53-4283-af3a-c0afd45dd841-kube-api-access-dlcw5\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.055947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.055921 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.056707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.056677 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.147248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.147149 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:03.157864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.157822 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.158015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.157878 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.158123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.158101 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcw5\" (UniqueName: \"kubernetes.io/projected/bccf430b-df53-4283-af3a-c0afd45dd841-kube-api-access-dlcw5\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.158485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.158459 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.160951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.160925 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.170605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.170307 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcw5\" (UniqueName: \"kubernetes.io/projected/bccf430b-df53-4283-af3a-c0afd45dd841-kube-api-access-dlcw5\") pod \"maas-api-revoke-sub-a504e80c-6dv26\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.264870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.264839 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:03.376321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.376288 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:03.449633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.449598 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85"] Nov 01 22:37:03.450460 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:03.450430 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d63c56_fb4c_4dcd_b000_59dd90860843.slice/crio-8c9b4b946ab43f0ccebb35131330ede5bc43d8fd829e6e5c547d060862d80d2c WatchSource:0}: Error finding container 8c9b4b946ab43f0ccebb35131330ede5bc43d8fd829e6e5c547d060862d80d2c: Status 404 returned error can't find the container with id 8c9b4b946ab43f0ccebb35131330ede5bc43d8fd829e6e5c547d060862d80d2c Nov 01 22:37:03.556059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.556035 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26"] Nov 01 22:37:03.556928 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:03.556901 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccf430b_df53_4283_af3a_c0afd45dd841.slice/crio-230a3f64ee1a97f8e8a0f0365bcfe91ba8b253e053c2515d94c988bb08d78a64 WatchSource:0}: Error finding container 230a3f64ee1a97f8e8a0f0365bcfe91ba8b253e053c2515d94c988bb08d78a64: Status 404 returned error can't find the container with id 230a3f64ee1a97f8e8a0f0365bcfe91ba8b253e053c2515d94c988bb08d78a64 Nov 01 22:37:03.947474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.947437 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" event={"ID":"38d63c56-fb4c-4dcd-b000-59dd90860843","Type":"ContainerStarted","Data":"3d07c46586a3a7ea2c21dac4ae12914a5a880ea2ea22c98033aec362bb43f74a"} Nov 01 22:37:03.947629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.947491 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" event={"ID":"38d63c56-fb4c-4dcd-b000-59dd90860843","Type":"ContainerStarted","Data":"8c9b4b946ab43f0ccebb35131330ede5bc43d8fd829e6e5c547d060862d80d2c"} Nov 01 22:37:03.949123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.949096 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" event={"ID":"bccf430b-df53-4283-af3a-c0afd45dd841","Type":"ContainerStarted","Data":"414db64fc857aeb2e8800c92cd899e5c3361116c53bacc7d9813639f7e3d6c88"} Nov 01 22:37:03.949218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.949135 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" event={"ID":"bccf430b-df53-4283-af3a-c0afd45dd841","Type":"ContainerStarted","Data":"230a3f64ee1a97f8e8a0f0365bcfe91ba8b253e053c2515d94c988bb08d78a64"} Nov 01 22:37:03.966699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.966651 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" podStartSLOduration=1.966639889 podStartE2EDuration="1.966639889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:03.964968839 +0000 UTC m=+789.392846770" watchObservedRunningTime="2026-11-01 22:37:03.966639889 +0000 UTC m=+789.394517820" Nov 01 22:37:03.983305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:03.983266 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" podStartSLOduration=1.9832533749999999 podStartE2EDuration="1.983253375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:03.981717306 +0000 UTC m=+789.409595238" watchObservedRunningTime="2026-11-01 22:37:03.983253375 +0000 UTC m=+789.411131305" Nov 01 22:37:04.073997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.073900 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:04.525649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.525621 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7"] Nov 01 22:37:04.529661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.529641 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.538601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.538546 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7"] Nov 01 22:37:04.575395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.575372 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.575515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.575433 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nz2\" (UniqueName: \"kubernetes.io/projected/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-kube-api-access-b5nz2\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.575599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.575512 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.676740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.676713 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.676890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.676869 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.676954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.676909 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nz2\" (UniqueName: \"kubernetes.io/projected/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-kube-api-access-b5nz2\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.677418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.677393 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.679222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.679201 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.686808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.686788 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nz2\" (UniqueName: \"kubernetes.io/projected/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-kube-api-access-b5nz2\") pod \"maas-api-revoke-sub-1642c62e-czpt7\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.790453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.790424 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw"] Nov 01 22:37:04.794537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.794520 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.801683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.801660 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw"] Nov 01 22:37:04.860033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.860003 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:04.879885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.879854 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv49\" (UniqueName: \"kubernetes.io/projected/66b14fec-8347-4ba2-87fe-6972567bac98-kube-api-access-4wv49\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.880039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.880018 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.880120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.880101 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.949462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.949382 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:04.959075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.959047 4981 generic.go:356] "Generic (PLEG): container finished" podID="38d63c56-fb4c-4dcd-b000-59dd90860843" containerID="3d07c46586a3a7ea2c21dac4ae12914a5a880ea2ea22c98033aec362bb43f74a" exitCode=0 Nov 01 22:37:04.959215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.959140 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" event={"ID":"38d63c56-fb4c-4dcd-b000-59dd90860843","Type":"ContainerDied","Data":"3d07c46586a3a7ea2c21dac4ae12914a5a880ea2ea22c98033aec362bb43f74a"} Nov 01 22:37:04.960899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.960878 4981 generic.go:356] "Generic (PLEG): container finished" podID="bccf430b-df53-4283-af3a-c0afd45dd841" containerID="414db64fc857aeb2e8800c92cd899e5c3361116c53bacc7d9813639f7e3d6c88" exitCode=0 Nov 01 22:37:04.961008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.960966 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" event={"ID":"bccf430b-df53-4283-af3a-c0afd45dd841","Type":"ContainerDied","Data":"414db64fc857aeb2e8800c92cd899e5c3361116c53bacc7d9813639f7e3d6c88"} Nov 01 22:37:04.981079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.981052 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv49\" (UniqueName: \"kubernetes.io/projected/66b14fec-8347-4ba2-87fe-6972567bac98-kube-api-access-4wv49\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.981201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.981137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.981268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.981200 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.982007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.981983 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.983869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.983846 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:04.990185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:04.990161 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv49\" (UniqueName: \"kubernetes.io/projected/66b14fec-8347-4ba2-87fe-6972567bac98-kube-api-access-4wv49\") pod \"maas-api-revoke-sub-e341f3ef-rgjdw\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:05.110697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.110661 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:05.148779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.148749 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7"] Nov 01 22:37:05.149536 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:05.149511 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3b7e13_11b1_4b30_99c7_7ea9c817a290.slice/crio-3b36c9b27a430217c0635bd53238f9c7b753a7235318301dccc897e529a126d5 WatchSource:0}: Error finding container 3b36c9b27a430217c0635bd53238f9c7b753a7235318301dccc897e529a126d5: Status 404 returned error can't find the container with id 3b36c9b27a430217c0635bd53238f9c7b753a7235318301dccc897e529a126d5 Nov 01 22:37:05.397140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.397109 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw"] Nov 01 22:37:05.397431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:05.397408 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b14fec_8347_4ba2_87fe_6972567bac98.slice/crio-e282d46ed1b9e15cf511e00dec2bc3aef87a6905c323edcff928f5ed6d5cd280 WatchSource:0}: Error finding container e282d46ed1b9e15cf511e00dec2bc3aef87a6905c323edcff928f5ed6d5cd280: Status 404 returned error can't find the container with id e282d46ed1b9e15cf511e00dec2bc3aef87a6905c323edcff928f5ed6d5cd280 Nov 01 22:37:05.759658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.759627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:05.969416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.969325 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" event={"ID":"9a3b7e13-11b1-4b30-99c7-7ea9c817a290","Type":"ContainerStarted","Data":"23c84757269d8fc3a8c5ef60acce0d5d1ae3a6c48a77b314a0ebd222a41b7811"} Nov 01 22:37:05.969416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.969386 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" event={"ID":"9a3b7e13-11b1-4b30-99c7-7ea9c817a290","Type":"ContainerStarted","Data":"3b36c9b27a430217c0635bd53238f9c7b753a7235318301dccc897e529a126d5"} Nov 01 22:37:05.971209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.971183 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" event={"ID":"66b14fec-8347-4ba2-87fe-6972567bac98","Type":"ContainerStarted","Data":"6e3afb8e84ee3fd78b928108fe2dcd641f63b05608cbdb89e678775f8f39129d"} Nov 01 22:37:05.971289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.971233 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" event={"ID":"66b14fec-8347-4ba2-87fe-6972567bac98","Type":"ContainerStarted","Data":"e282d46ed1b9e15cf511e00dec2bc3aef87a6905c323edcff928f5ed6d5cd280"} Nov 01 22:37:05.989559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:05.989498 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" podStartSLOduration=1.989484773 podStartE2EDuration="1.989484773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:05.986095426 +0000 UTC m=+791.413973362" watchObservedRunningTime="2026-11-01 22:37:05.989484773 +0000 UTC m=+791.417362737" Nov 01 22:37:06.010610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.008539 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" podStartSLOduration=2.0085227 podStartE2EDuration="2.0085227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:06.004912601 +0000 UTC m=+791.432790534" watchObservedRunningTime="2026-11-01 22:37:06.0085227 +0000 UTC m=+791.436400629" Nov 01 22:37:06.136470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.136440 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:06.158047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.158014 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:06.198008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.197972 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-xks-ca\") pod \"38d63c56-fb4c-4dcd-b000-59dd90860843\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " Nov 01 22:37:06.198189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198030 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d63c56-fb4c-4dcd-b000-59dd90860843-kube-api-access-tmpn7\" (UniqueName: \"kubernetes.io/projected/38d63c56-fb4c-4dcd-b000-59dd90860843-kube-api-access-tmpn7\") pod \"38d63c56-fb4c-4dcd-b000-59dd90860843\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " Nov 01 22:37:06.198189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198141 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-xks-ca\") pod \"bccf430b-df53-4283-af3a-c0afd45dd841\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " Nov 01 22:37:06.198189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198184 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-openshift-ca\") pod \"bccf430b-df53-4283-af3a-c0afd45dd841\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " Nov 01 22:37:06.198377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198286 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccf430b-df53-4283-af3a-c0afd45dd841-kube-api-access-dlcw5\" (UniqueName: \"kubernetes.io/projected/bccf430b-df53-4283-af3a-c0afd45dd841-kube-api-access-dlcw5\") pod \"bccf430b-df53-4283-af3a-c0afd45dd841\" (UID: \"bccf430b-df53-4283-af3a-c0afd45dd841\") " Nov 01 22:37:06.198377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198324 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-openshift-ca\") pod \"38d63c56-fb4c-4dcd-b000-59dd90860843\" (UID: \"38d63c56-fb4c-4dcd-b000-59dd90860843\") " Nov 01 22:37:06.199041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198653 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-openshift-ca" pod "bccf430b-df53-4283-af3a-c0afd45dd841" (UID: "bccf430b-df53-4283-af3a-c0afd45dd841"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:06.199041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198857 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-openshift-ca" pod "38d63c56-fb4c-4dcd-b000-59dd90860843" (UID: "38d63c56-fb4c-4dcd-b000-59dd90860843"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:06.199041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.198984 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.199041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.199005 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.200541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.200517 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-xks-ca" pod "38d63c56-fb4c-4dcd-b000-59dd90860843" (UID: "38d63c56-fb4c-4dcd-b000-59dd90860843"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:06.201013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.200926 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-xks-ca" pod "bccf430b-df53-4283-af3a-c0afd45dd841" (UID: "bccf430b-df53-4283-af3a-c0afd45dd841"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:06.201013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.200976 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccf430b-df53-4283-af3a-c0afd45dd841-kube-api-access-dlcw5" pod "bccf430b-df53-4283-af3a-c0afd45dd841" (UID: "bccf430b-df53-4283-af3a-c0afd45dd841"). InnerVolumeSpecName "kube-api-access-dlcw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:06.201123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.201037 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d63c56-fb4c-4dcd-b000-59dd90860843-kube-api-access-tmpn7" pod "38d63c56-fb4c-4dcd-b000-59dd90860843" (UID: "38d63c56-fb4c-4dcd-b000-59dd90860843"). InnerVolumeSpecName "kube-api-access-tmpn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:06.299838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.299805 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d63c56-fb4c-4dcd-b000-59dd90860843-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.299947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.299843 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmpn7\" (UniqueName: \"kubernetes.io/projected/38d63c56-fb4c-4dcd-b000-59dd90860843-kube-api-access-tmpn7\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.299947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.299859 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf430b-df53-4283-af3a-c0afd45dd841-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.299947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.299873 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlcw5\" (UniqueName: \"kubernetes.io/projected/bccf430b-df53-4283-af3a-c0afd45dd841-kube-api-access-dlcw5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:06.566842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.566756 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:06.889951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.889909 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq"] Nov 01 22:37:06.890521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.890506 4981 cpu_manager.go:414] "containerMap: removing container" podUID="38d63c56-fb4c-4dcd-b000-59dd90860843" containerName="revoke-keys" Nov 01 22:37:06.890568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.890523 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d63c56-fb4c-4dcd-b000-59dd90860843" containerName="revoke-keys" Nov 01 22:37:06.890568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.890543 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bccf430b-df53-4283-af3a-c0afd45dd841" containerName="revoke-keys" Nov 01 22:37:06.890568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.890549 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccf430b-df53-4283-af3a-c0afd45dd841" containerName="revoke-keys" Nov 01 22:37:06.890683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.890671 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d63c56-fb4c-4dcd-b000-59dd90860843" containerName="revoke-keys" Nov 01 22:37:06.890719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.890683 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccf430b-df53-4283-af3a-c0afd45dd841" containerName="revoke-keys" Nov 01 22:37:06.894185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.894163 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:06.896908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.896887 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m"] Nov 01 22:37:06.901319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.900975 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:06.904876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.904857 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq"] Nov 01 22:37:06.919189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.919167 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m"] Nov 01 22:37:06.979832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.979797 4981 generic.go:356] "Generic (PLEG): container finished" podID="9a3b7e13-11b1-4b30-99c7-7ea9c817a290" containerID="23c84757269d8fc3a8c5ef60acce0d5d1ae3a6c48a77b314a0ebd222a41b7811" exitCode=0 Nov 01 22:37:06.980012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.979842 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" event={"ID":"9a3b7e13-11b1-4b30-99c7-7ea9c817a290","Type":"ContainerDied","Data":"23c84757269d8fc3a8c5ef60acce0d5d1ae3a6c48a77b314a0ebd222a41b7811"} Nov 01 22:37:06.981171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.981155 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" Nov 01 22:37:06.981236 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.981155 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85" event={"ID":"38d63c56-fb4c-4dcd-b000-59dd90860843","Type":"ContainerDied","Data":"8c9b4b946ab43f0ccebb35131330ede5bc43d8fd829e6e5c547d060862d80d2c"} Nov 01 22:37:06.981280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.981264 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c9b4b946ab43f0ccebb35131330ede5bc43d8fd829e6e5c547d060862d80d2c" Nov 01 22:37:06.982685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.982663 4981 generic.go:356] "Generic (PLEG): container finished" podID="66b14fec-8347-4ba2-87fe-6972567bac98" containerID="6e3afb8e84ee3fd78b928108fe2dcd641f63b05608cbdb89e678775f8f39129d" exitCode=0 Nov 01 22:37:06.982801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.982735 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" event={"ID":"66b14fec-8347-4ba2-87fe-6972567bac98","Type":"ContainerDied","Data":"6e3afb8e84ee3fd78b928108fe2dcd641f63b05608cbdb89e678775f8f39129d"} Nov 01 22:37:06.984151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.984132 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" event={"ID":"bccf430b-df53-4283-af3a-c0afd45dd841","Type":"ContainerDied","Data":"230a3f64ee1a97f8e8a0f0365bcfe91ba8b253e053c2515d94c988bb08d78a64"} Nov 01 22:37:06.984235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.984164 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="230a3f64ee1a97f8e8a0f0365bcfe91ba8b253e053c2515d94c988bb08d78a64" Nov 01 22:37:06.984235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:06.984166 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26" Nov 01 22:37:07.007339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.007314 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.007466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.007365 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjwg\" (UniqueName: \"kubernetes.io/projected/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-kube-api-access-2gjwg\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.007466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.007457 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.007566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.007487 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.007710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.007688 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.007858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.007739 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mpq\" (UniqueName: \"kubernetes.io/projected/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-kube-api-access-r6mpq\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.109325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.109292 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.109485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.109355 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjwg\" (UniqueName: \"kubernetes.io/projected/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-kube-api-access-2gjwg\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.109485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.109393 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.109485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.109414 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.109485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.109473 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.109668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.109490 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mpq\" (UniqueName: \"kubernetes.io/projected/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-kube-api-access-r6mpq\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.110112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.110085 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.110246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.110140 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.111722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.111693 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.111925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.111905 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.117762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.117744 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mpq\" (UniqueName: \"kubernetes.io/projected/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-kube-api-access-r6mpq\") pod \"maas-api-revoke-sub-f9e8db3e-sc8cq\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.117899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.117879 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjwg\" (UniqueName: \"kubernetes.io/projected/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-kube-api-access-2gjwg\") pod \"maas-api-revoke-sub-f088f6ba-55k9m\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.212691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.212669 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:07.223249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.223226 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:07.379001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.378957 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:07.561758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.561725 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq"] Nov 01 22:37:07.562816 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:07.562792 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a4bc7e_0f36_4c30_98e4_d5342ee24b79.slice/crio-87744f42af08c7656051ae76158e29f5950969e4d4d5d8573de5ef9752cb17c5 WatchSource:0}: Error finding container 87744f42af08c7656051ae76158e29f5950969e4d4d5d8573de5ef9752cb17c5: Status 404 returned error can't find the container with id 87744f42af08c7656051ae76158e29f5950969e4d4d5d8573de5ef9752cb17c5 Nov 01 22:37:07.569604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.569566 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m"] Nov 01 22:37:07.574787 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:07.574761 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c4bf50_afa1_42ed_b79b_d4e24aaf9196.slice/crio-2e95dfef1c69e675aa8c91c84cdb919db96243d8b790c38ac941fa07a048d7fa WatchSource:0}: Error finding container 2e95dfef1c69e675aa8c91c84cdb919db96243d8b790c38ac941fa07a048d7fa: Status 404 returned error can't find the container with id 2e95dfef1c69e675aa8c91c84cdb919db96243d8b790c38ac941fa07a048d7fa Nov 01 22:37:07.992513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.992488 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" event={"ID":"43c4bf50-afa1-42ed-b79b-d4e24aaf9196","Type":"ContainerStarted","Data":"2e95dfef1c69e675aa8c91c84cdb919db96243d8b790c38ac941fa07a048d7fa"} Nov 01 22:37:07.993979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.993954 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" event={"ID":"67a4bc7e-0f36-4c30-98e4-d5342ee24b79","Type":"ContainerStarted","Data":"1121fac32c5a4821630fd8a9975e714e1de30b238a9c3c4e1c4d6b0a9c19f218"} Nov 01 22:37:07.994069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:07.994003 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" event={"ID":"67a4bc7e-0f36-4c30-98e4-d5342ee24b79","Type":"ContainerStarted","Data":"87744f42af08c7656051ae76158e29f5950969e4d4d5d8573de5ef9752cb17c5"} Nov 01 22:37:08.150337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.150304 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:08.178891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.178867 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:08.182979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.182958 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:08.200046 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.199997 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" podStartSLOduration=2.199981585 podStartE2EDuration="2.199981585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:08.014648478 +0000 UTC m=+793.442526408" watchObservedRunningTime="2026-11-01 22:37:08.199981585 +0000 UTC m=+793.627859517" Nov 01 22:37:08.222743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.222695 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-xks-ca\") pod \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " Nov 01 22:37:08.223733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.223712 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b14fec-8347-4ba2-87fe-6972567bac98-kube-api-access-4wv49\" (UniqueName: \"kubernetes.io/projected/66b14fec-8347-4ba2-87fe-6972567bac98-kube-api-access-4wv49\") pod \"66b14fec-8347-4ba2-87fe-6972567bac98\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " Nov 01 22:37:08.224217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.224198 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-kube-api-access-b5nz2\" (UniqueName: \"kubernetes.io/projected/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-kube-api-access-b5nz2\") pod \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " Nov 01 22:37:08.224379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.224365 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-xks-ca\") pod \"66b14fec-8347-4ba2-87fe-6972567bac98\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " Nov 01 22:37:08.224487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.224475 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-openshift-ca\") pod \"66b14fec-8347-4ba2-87fe-6972567bac98\" (UID: \"66b14fec-8347-4ba2-87fe-6972567bac98\") " Nov 01 22:37:08.224643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.224628 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-openshift-ca\") pod \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\" (UID: \"9a3b7e13-11b1-4b30-99c7-7ea9c817a290\") " Nov 01 22:37:08.225352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.225302 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-openshift-ca" pod "66b14fec-8347-4ba2-87fe-6972567bac98" (UID: "66b14fec-8347-4ba2-87fe-6972567bac98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:08.225886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.225542 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-openshift-ca" pod "9a3b7e13-11b1-4b30-99c7-7ea9c817a290" (UID: "9a3b7e13-11b1-4b30-99c7-7ea9c817a290"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:08.228342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.228318 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-xks-ca" pod "66b14fec-8347-4ba2-87fe-6972567bac98" (UID: "66b14fec-8347-4ba2-87fe-6972567bac98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:08.237005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.236967 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-kube-api-access-b5nz2" pod "9a3b7e13-11b1-4b30-99c7-7ea9c817a290" (UID: "9a3b7e13-11b1-4b30-99c7-7ea9c817a290"). InnerVolumeSpecName "kube-api-access-b5nz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:08.237110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.237093 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b14fec-8347-4ba2-87fe-6972567bac98-kube-api-access-4wv49" pod "66b14fec-8347-4ba2-87fe-6972567bac98" (UID: "66b14fec-8347-4ba2-87fe-6972567bac98"). InnerVolumeSpecName "kube-api-access-4wv49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:08.237653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.237623 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-xks-ca" pod "9a3b7e13-11b1-4b30-99c7-7ea9c817a290" (UID: "9a3b7e13-11b1-4b30-99c7-7ea9c817a290"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:08.326033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.326002 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:08.326033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.326029 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:08.326033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.326039 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wv49\" (UniqueName: \"kubernetes.io/projected/66b14fec-8347-4ba2-87fe-6972567bac98-kube-api-access-4wv49\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:08.326238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.326048 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5nz2\" (UniqueName: \"kubernetes.io/projected/9a3b7e13-11b1-4b30-99c7-7ea9c817a290-kube-api-access-b5nz2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:08.326238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.326058 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:08.326238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.326067 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66b14fec-8347-4ba2-87fe-6972567bac98-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:08.866728 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:08.866692 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:09.001660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.001555 4981 generic.go:356] "Generic (PLEG): container finished" podID="43c4bf50-afa1-42ed-b79b-d4e24aaf9196" containerID="6f3ca318211b0807c27f834daaef99cd0a663c303f3ebe089c19575598169697" exitCode=0 Nov 01 22:37:09.001660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.001643 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" event={"ID":"43c4bf50-afa1-42ed-b79b-d4e24aaf9196","Type":"ContainerDied","Data":"6f3ca318211b0807c27f834daaef99cd0a663c303f3ebe089c19575598169697"} Nov 01 22:37:09.002922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.002904 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" Nov 01 22:37:09.003016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.002936 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw" event={"ID":"66b14fec-8347-4ba2-87fe-6972567bac98","Type":"ContainerDied","Data":"e282d46ed1b9e15cf511e00dec2bc3aef87a6905c323edcff928f5ed6d5cd280"} Nov 01 22:37:09.003016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.002974 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e282d46ed1b9e15cf511e00dec2bc3aef87a6905c323edcff928f5ed6d5cd280" Nov 01 22:37:09.004408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.004384 4981 generic.go:356] "Generic (PLEG): container finished" podID="67a4bc7e-0f36-4c30-98e4-d5342ee24b79" containerID="1121fac32c5a4821630fd8a9975e714e1de30b238a9c3c4e1c4d6b0a9c19f218" exitCode=0 Nov 01 22:37:09.004513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.004463 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" event={"ID":"67a4bc7e-0f36-4c30-98e4-d5342ee24b79","Type":"ContainerDied","Data":"1121fac32c5a4821630fd8a9975e714e1de30b238a9c3c4e1c4d6b0a9c19f218"} Nov 01 22:37:09.006138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.006117 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" event={"ID":"9a3b7e13-11b1-4b30-99c7-7ea9c817a290","Type":"ContainerDied","Data":"3b36c9b27a430217c0635bd53238f9c7b753a7235318301dccc897e529a126d5"} Nov 01 22:37:09.006235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.006159 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b36c9b27a430217c0635bd53238f9c7b753a7235318301dccc897e529a126d5" Nov 01 22:37:09.006235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.006124 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7" Nov 01 22:37:09.658376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:09.658334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:10.173097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.173067 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:10.218051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.218025 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:10.245836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.245802 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-kube-api-access-r6mpq\" (UniqueName: \"kubernetes.io/projected/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-kube-api-access-r6mpq\") pod \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " Nov 01 22:37:10.245993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.245879 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-xks-ca\") pod \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " Nov 01 22:37:10.245993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.245925 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-xks-ca\") pod \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " Nov 01 22:37:10.246090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.246072 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-openshift-ca\") pod \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " Nov 01 22:37:10.246184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.246112 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-openshift-ca\") pod \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\" (UID: \"67a4bc7e-0f36-4c30-98e4-d5342ee24b79\") " Nov 01 22:37:10.246242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.246184 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-kube-api-access-2gjwg\" (UniqueName: \"kubernetes.io/projected/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-kube-api-access-2gjwg\") pod \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\" (UID: \"43c4bf50-afa1-42ed-b79b-d4e24aaf9196\") " Nov 01 22:37:10.246779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.246437 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-openshift-ca" pod "43c4bf50-afa1-42ed-b79b-d4e24aaf9196" (UID: "43c4bf50-afa1-42ed-b79b-d4e24aaf9196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:10.246779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.246723 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-openshift-ca" pod "67a4bc7e-0f36-4c30-98e4-d5342ee24b79" (UID: "67a4bc7e-0f36-4c30-98e4-d5342ee24b79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:10.246779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.246735 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:10.248495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.248464 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-kube-api-access-2gjwg" pod "43c4bf50-afa1-42ed-b79b-d4e24aaf9196" (UID: "43c4bf50-afa1-42ed-b79b-d4e24aaf9196"). InnerVolumeSpecName "kube-api-access-2gjwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:10.248751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.248732 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-kube-api-access-r6mpq" pod "67a4bc7e-0f36-4c30-98e4-d5342ee24b79" (UID: "67a4bc7e-0f36-4c30-98e4-d5342ee24b79"). InnerVolumeSpecName "kube-api-access-r6mpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:10.256621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.256595 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-xks-ca" pod "43c4bf50-afa1-42ed-b79b-d4e24aaf9196" (UID: "43c4bf50-afa1-42ed-b79b-d4e24aaf9196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:10.257158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.257134 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-xks-ca" pod "67a4bc7e-0f36-4c30-98e4-d5342ee24b79" (UID: "67a4bc7e-0f36-4c30-98e4-d5342ee24b79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:10.345104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.345018 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:10.347410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.347389 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6mpq\" (UniqueName: \"kubernetes.io/projected/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-kube-api-access-r6mpq\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:10.347509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.347411 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:10.347509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.347426 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:10.347509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.347440 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a4bc7e-0f36-4c30-98e4-d5342ee24b79-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:10.347509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:10.347454 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjwg\" (UniqueName: \"kubernetes.io/projected/43c4bf50-afa1-42ed-b79b-d4e24aaf9196-kube-api-access-2gjwg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:11.019678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:11.019563 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" Nov 01 22:37:11.019678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:11.019615 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m" event={"ID":"43c4bf50-afa1-42ed-b79b-d4e24aaf9196","Type":"ContainerDied","Data":"2e95dfef1c69e675aa8c91c84cdb919db96243d8b790c38ac941fa07a048d7fa"} Nov 01 22:37:11.019678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:11.019667 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e95dfef1c69e675aa8c91c84cdb919db96243d8b790c38ac941fa07a048d7fa" Nov 01 22:37:11.021286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:11.021267 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" Nov 01 22:37:11.021436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:11.021266 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq" event={"ID":"67a4bc7e-0f36-4c30-98e4-d5342ee24b79","Type":"ContainerDied","Data":"87744f42af08c7656051ae76158e29f5950969e4d4d5d8573de5ef9752cb17c5"} Nov 01 22:37:11.021436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:11.021397 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87744f42af08c7656051ae76158e29f5950969e4d4d5d8573de5ef9752cb17c5" Nov 01 22:37:11.074975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:11.074944 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:12.121598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:12.121155 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:12.332597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:12.332146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:13.315600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:13.314643 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:14.156703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:14.156652 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:14.290491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:14.290445 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:14.622014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:14.621981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:14.641257 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:37:14.641220 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:37:15.229506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.229473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:15.503232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.503152 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6"] Nov 01 22:37:15.503982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.503964 4981 cpu_manager.go:414] "containerMap: removing container" podUID="43c4bf50-afa1-42ed-b79b-d4e24aaf9196" containerName="revoke-keys" Nov 01 22:37:15.504031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.503987 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c4bf50-afa1-42ed-b79b-d4e24aaf9196" containerName="revoke-keys" Nov 01 22:37:15.504031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504017 4981 cpu_manager.go:414] "containerMap: removing container" podUID="66b14fec-8347-4ba2-87fe-6972567bac98" containerName="revoke-keys" Nov 01 22:37:15.504031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504025 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b14fec-8347-4ba2-87fe-6972567bac98" containerName="revoke-keys" Nov 01 22:37:15.504155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504082 4981 cpu_manager.go:414] "containerMap: removing container" podUID="67a4bc7e-0f36-4c30-98e4-d5342ee24b79" containerName="revoke-keys" Nov 01 22:37:15.504155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504090 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a4bc7e-0f36-4c30-98e4-d5342ee24b79" containerName="revoke-keys" Nov 01 22:37:15.504155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504105 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9a3b7e13-11b1-4b30-99c7-7ea9c817a290" containerName="revoke-keys" Nov 01 22:37:15.504155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504113 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3b7e13-11b1-4b30-99c7-7ea9c817a290" containerName="revoke-keys" Nov 01 22:37:15.504329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504224 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a4bc7e-0f36-4c30-98e4-d5342ee24b79" containerName="revoke-keys" Nov 01 22:37:15.504329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504234 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3b7e13-11b1-4b30-99c7-7ea9c817a290" containerName="revoke-keys" Nov 01 22:37:15.504329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504258 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c4bf50-afa1-42ed-b79b-d4e24aaf9196" containerName="revoke-keys" Nov 01 22:37:15.504329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.504267 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b14fec-8347-4ba2-87fe-6972567bac98" containerName="revoke-keys" Nov 01 22:37:15.508703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.508680 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.514679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.514158 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:15.514679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.514187 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:15.514679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.514189 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:15.514679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.514254 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:37:15.517543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.517516 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6"] Nov 01 22:37:15.552695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.552658 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.552888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.552862 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.552987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.552926 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzczw\" (UniqueName: \"kubernetes.io/projected/dc6cdf03-19a1-435a-9268-a17091fdbc01-kube-api-access-xzczw\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.654679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.654642 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.655130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.654802 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.655130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.654866 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzczw\" (UniqueName: \"kubernetes.io/projected/dc6cdf03-19a1-435a-9268-a17091fdbc01-kube-api-access-xzczw\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.655542 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.655517 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.657070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.657051 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.663507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.663483 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzczw\" (UniqueName: \"kubernetes.io/projected/dc6cdf03-19a1-435a-9268-a17091fdbc01-kube-api-access-xzczw\") pod \"maas-api-revoke-sub-573d542f-r88v6\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.823385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.823307 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:15.850731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.850688 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:15.938527 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:15.938491 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:16.108323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:16.108294 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6"] Nov 01 22:37:16.108988 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:16.108963 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6cdf03_19a1_435a_9268_a17091fdbc01.slice/crio-0fa996ff5f9f78ee27b1a007b52cd68acfdce9299d97178616b1083742783f4a WatchSource:0}: Error finding container 0fa996ff5f9f78ee27b1a007b52cd68acfdce9299d97178616b1083742783f4a: Status 404 returned error can't find the container with id 0fa996ff5f9f78ee27b1a007b52cd68acfdce9299d97178616b1083742783f4a Nov 01 22:37:16.759495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:16.759460 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:17.091778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.091702 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9"] Nov 01 22:37:17.095923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.095898 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.097451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.097425 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc6cdf03-19a1-435a-9268-a17091fdbc01" containerID="2a6ecef2d99625b1526f5025f039dae3db86f0aad3a9da331587d67da895bdf2" exitCode=0 Nov 01 22:37:17.097601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.097456 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" event={"ID":"dc6cdf03-19a1-435a-9268-a17091fdbc01","Type":"ContainerDied","Data":"2a6ecef2d99625b1526f5025f039dae3db86f0aad3a9da331587d67da895bdf2"} Nov 01 22:37:17.097601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.097508 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" event={"ID":"dc6cdf03-19a1-435a-9268-a17091fdbc01","Type":"ContainerStarted","Data":"0fa996ff5f9f78ee27b1a007b52cd68acfdce9299d97178616b1083742783f4a"} Nov 01 22:37:17.107764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.107738 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9"] Nov 01 22:37:17.172570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.172533 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hvz\" (UniqueName: \"kubernetes.io/projected/dc47d047-7b84-49b0-9096-dc2b9b9b8989-kube-api-access-84hvz\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.172777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.172680 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.172777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.172750 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.273697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.273660 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hvz\" (UniqueName: \"kubernetes.io/projected/dc47d047-7b84-49b0-9096-dc2b9b9b8989-kube-api-access-84hvz\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.273890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.273736 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.273890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.273790 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.274543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.274514 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.276220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.276193 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.286780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.286747 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hvz\" (UniqueName: \"kubernetes.io/projected/dc47d047-7b84-49b0-9096-dc2b9b9b8989-kube-api-access-84hvz\") pod \"maas-api-revoke-sub-8ba2d1b5-dvpx9\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.414488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.414403 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:17.649142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.649081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:17.698537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:17.698491 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9"] Nov 01 22:37:17.700982 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:17.700349 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc47d047_7b84_49b0_9096_dc2b9b9b8989.slice/crio-cf517e96d9169860d233c0c5b581dce6116a693cd243dd410f77be69e48d5482 WatchSource:0}: Error finding container cf517e96d9169860d233c0c5b581dce6116a693cd243dd410f77be69e48d5482: Status 404 returned error can't find the container with id cf517e96d9169860d233c0c5b581dce6116a693cd243dd410f77be69e48d5482 Nov 01 22:37:18.109389 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.109351 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" event={"ID":"dc47d047-7b84-49b0-9096-dc2b9b9b8989","Type":"ContainerStarted","Data":"1931426763ac9c25cf1f1132cb3dfbc26b2f4a4d3808fa132d5501b8b16d9732"} Nov 01 22:37:18.109863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.109411 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" event={"ID":"dc47d047-7b84-49b0-9096-dc2b9b9b8989","Type":"ContainerStarted","Data":"cf517e96d9169860d233c0c5b581dce6116a693cd243dd410f77be69e48d5482"} Nov 01 22:37:18.165301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.165234 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" podStartSLOduration=1.165214297 podStartE2EDuration="1.165214297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:18.160491965 +0000 UTC m=+803.588369900" watchObservedRunningTime="2026-11-01 22:37:18.165214297 +0000 UTC m=+803.593092229" Nov 01 22:37:18.264874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.264845 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:18.286375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.286342 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-openshift-ca\") pod \"dc6cdf03-19a1-435a-9268-a17091fdbc01\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " Nov 01 22:37:18.286594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.286395 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-xks-ca\") pod \"dc6cdf03-19a1-435a-9268-a17091fdbc01\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " Nov 01 22:37:18.286594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.286488 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6cdf03-19a1-435a-9268-a17091fdbc01-kube-api-access-xzczw\" (UniqueName: \"kubernetes.io/projected/dc6cdf03-19a1-435a-9268-a17091fdbc01-kube-api-access-xzczw\") pod \"dc6cdf03-19a1-435a-9268-a17091fdbc01\" (UID: \"dc6cdf03-19a1-435a-9268-a17091fdbc01\") " Nov 01 22:37:18.286887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.286856 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-openshift-ca" pod "dc6cdf03-19a1-435a-9268-a17091fdbc01" (UID: "dc6cdf03-19a1-435a-9268-a17091fdbc01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:18.287148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.287117 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:18.289328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.289298 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6cdf03-19a1-435a-9268-a17091fdbc01-kube-api-access-xzczw" pod "dc6cdf03-19a1-435a-9268-a17091fdbc01" (UID: "dc6cdf03-19a1-435a-9268-a17091fdbc01"). InnerVolumeSpecName "kube-api-access-xzczw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:18.289649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.289623 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-xks-ca" pod "dc6cdf03-19a1-435a-9268-a17091fdbc01" (UID: "dc6cdf03-19a1-435a-9268-a17091fdbc01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:18.387933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.387888 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc6cdf03-19a1-435a-9268-a17091fdbc01-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:18.387933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.387931 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzczw\" (UniqueName: \"kubernetes.io/projected/dc6cdf03-19a1-435a-9268-a17091fdbc01-kube-api-access-xzczw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:18.789529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.789451 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b"] Nov 01 22:37:18.790036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.790021 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc6cdf03-19a1-435a-9268-a17091fdbc01" containerName="revoke-keys" Nov 01 22:37:18.790076 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.790037 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6cdf03-19a1-435a-9268-a17091fdbc01" containerName="revoke-keys" Nov 01 22:37:18.790136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.790127 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6cdf03-19a1-435a-9268-a17091fdbc01" containerName="revoke-keys" Nov 01 22:37:18.793497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.793477 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.802786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.802758 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b"] Nov 01 22:37:18.827520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.827482 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:18.892959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.892924 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsjfm\" (UniqueName: \"kubernetes.io/projected/220066e3-6f27-44c8-bd95-843708b3095e-kube-api-access-rsjfm\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.893148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.893128 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.893195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.893154 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.994508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.994472 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsjfm\" (UniqueName: \"kubernetes.io/projected/220066e3-6f27-44c8-bd95-843708b3095e-kube-api-access-rsjfm\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.994696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.994638 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.994696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.994657 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.995259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.995241 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:18.997129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:18.997098 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:19.014013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.013984 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsjfm\" (UniqueName: \"kubernetes.io/projected/220066e3-6f27-44c8-bd95-843708b3095e-kube-api-access-rsjfm\") pod \"maas-api-revoke-sub-381dcaf7-q2r6b\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:19.109111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.109026 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:19.118480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.118450 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc47d047-7b84-49b0-9096-dc2b9b9b8989" containerID="1931426763ac9c25cf1f1132cb3dfbc26b2f4a4d3808fa132d5501b8b16d9732" exitCode=0 Nov 01 22:37:19.118895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.118542 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" event={"ID":"dc47d047-7b84-49b0-9096-dc2b9b9b8989","Type":"ContainerDied","Data":"1931426763ac9c25cf1f1132cb3dfbc26b2f4a4d3808fa132d5501b8b16d9732"} Nov 01 22:37:19.120082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.120064 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" Nov 01 22:37:19.120194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.120104 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6" event={"ID":"dc6cdf03-19a1-435a-9268-a17091fdbc01","Type":"ContainerDied","Data":"0fa996ff5f9f78ee27b1a007b52cd68acfdce9299d97178616b1083742783f4a"} Nov 01 22:37:19.120194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.120148 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fa996ff5f9f78ee27b1a007b52cd68acfdce9299d97178616b1083742783f4a" Nov 01 22:37:19.242000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.241960 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr"] Nov 01 22:37:19.262654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.262624 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.265464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.265434 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr"] Nov 01 22:37:19.297379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.297341 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzc4\" (UniqueName: \"kubernetes.io/projected/63ed81d4-7c21-415f-b409-1aee30f3123e-kube-api-access-ttzc4\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.297608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.297488 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.297666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.297624 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.397164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.397128 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b"] Nov 01 22:37:19.398553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.398525 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.398553 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:19.398542 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220066e3_6f27_44c8_bd95_843708b3095e.slice/crio-28d055d57d9381e7914c2a1422c7cc2e93513ba4a0e616cc21093f30863b0a6f WatchSource:0}: Error finding container 28d055d57d9381e7914c2a1422c7cc2e93513ba4a0e616cc21093f30863b0a6f: Status 404 returned error can't find the container with id 28d055d57d9381e7914c2a1422c7cc2e93513ba4a0e616cc21093f30863b0a6f Nov 01 22:37:19.398805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.398762 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzc4\" (UniqueName: \"kubernetes.io/projected/63ed81d4-7c21-415f-b409-1aee30f3123e-kube-api-access-ttzc4\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.398912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.398890 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.399386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.399357 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.402213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.402184 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.418185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.418154 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzc4\" (UniqueName: \"kubernetes.io/projected/63ed81d4-7c21-415f-b409-1aee30f3123e-kube-api-access-ttzc4\") pod \"maas-api-revoke-sub-06196609-9l9hr\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.586144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.586113 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:19.595278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.595251 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7"] Nov 01 22:37:19.600078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.600052 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.634924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.634869 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7"] Nov 01 22:37:19.701442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.701401 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.701642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.701596 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.701706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.701672 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxngt\" (UniqueName: \"kubernetes.io/projected/32810abe-3631-417c-89ba-da7a149a9abe-kube-api-access-cxngt\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.802631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.802563 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.802785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.802758 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.802844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.802817 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxngt\" (UniqueName: \"kubernetes.io/projected/32810abe-3631-417c-89ba-da7a149a9abe-kube-api-access-cxngt\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.803317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.803296 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.805154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.805133 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.813569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.813538 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxngt\" (UniqueName: \"kubernetes.io/projected/32810abe-3631-417c-89ba-da7a149a9abe-kube-api-access-cxngt\") pod \"maas-api-revoke-sub-5af2843b-44fs7\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:19.821516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.821489 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:19.890427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.890314 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr"] Nov 01 22:37:19.893406 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:19.893376 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ed81d4_7c21_415f_b409_1aee30f3123e.slice/crio-1d4e7bfe41404939e395ad771f09066808e27d9858fdcd65ffd59ffc7230c97c WatchSource:0}: Error finding container 1d4e7bfe41404939e395ad771f09066808e27d9858fdcd65ffd59ffc7230c97c: Status 404 returned error can't find the container with id 1d4e7bfe41404939e395ad771f09066808e27d9858fdcd65ffd59ffc7230c97c Nov 01 22:37:19.915707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:19.915679 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:20.128449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.128407 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" event={"ID":"63ed81d4-7c21-415f-b409-1aee30f3123e","Type":"ContainerStarted","Data":"1d4e7bfe41404939e395ad771f09066808e27d9858fdcd65ffd59ffc7230c97c"} Nov 01 22:37:20.130014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.129987 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" event={"ID":"220066e3-6f27-44c8-bd95-843708b3095e","Type":"ContainerStarted","Data":"c2313e74d0c10b68e34a2c8d7407debb0097c30e988da22fbdbbf2b1c1c65f8c"} Nov 01 22:37:20.130171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.130033 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" event={"ID":"220066e3-6f27-44c8-bd95-843708b3095e","Type":"ContainerStarted","Data":"28d055d57d9381e7914c2a1422c7cc2e93513ba4a0e616cc21093f30863b0a6f"} Nov 01 22:37:20.192203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.192155 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" podStartSLOduration=2.192138341 podStartE2EDuration="2.192138341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:20.147652052 +0000 UTC m=+805.575529982" watchObservedRunningTime="2026-11-01 22:37:20.192138341 +0000 UTC m=+805.620016271" Nov 01 22:37:20.194955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.194928 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7"] Nov 01 22:37:20.196335 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:20.196272 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32810abe_3631_417c_89ba_da7a149a9abe.slice/crio-0004935ff0431b0a9596a209c1341c14cc6844bf193d02a9cde11879eae404b4 WatchSource:0}: Error finding container 0004935ff0431b0a9596a209c1341c14cc6844bf193d02a9cde11879eae404b4: Status 404 returned error can't find the container with id 0004935ff0431b0a9596a209c1341c14cc6844bf193d02a9cde11879eae404b4 Nov 01 22:37:20.274147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.274118 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:20.310715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.310686 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc47d047-7b84-49b0-9096-dc2b9b9b8989-kube-api-access-84hvz\" (UniqueName: \"kubernetes.io/projected/dc47d047-7b84-49b0-9096-dc2b9b9b8989-kube-api-access-84hvz\") pod \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " Nov 01 22:37:20.310897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.310866 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-xks-ca\") pod \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " Nov 01 22:37:20.310977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.310961 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-openshift-ca\") pod \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\" (UID: \"dc47d047-7b84-49b0-9096-dc2b9b9b8989\") " Nov 01 22:37:20.311321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.311293 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-openshift-ca" pod "dc47d047-7b84-49b0-9096-dc2b9b9b8989" (UID: "dc47d047-7b84-49b0-9096-dc2b9b9b8989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:20.311498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.311477 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:20.312892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.312868 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc47d047-7b84-49b0-9096-dc2b9b9b8989-kube-api-access-84hvz" pod "dc47d047-7b84-49b0-9096-dc2b9b9b8989" (UID: "dc47d047-7b84-49b0-9096-dc2b9b9b8989"). InnerVolumeSpecName "kube-api-access-84hvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:20.313428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.313407 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-xks-ca" pod "dc47d047-7b84-49b0-9096-dc2b9b9b8989" (UID: "dc47d047-7b84-49b0-9096-dc2b9b9b8989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:20.413030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.412995 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hvz\" (UniqueName: \"kubernetes.io/projected/dc47d047-7b84-49b0-9096-dc2b9b9b8989-kube-api-access-84hvz\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:20.413030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.413028 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc47d047-7b84-49b0-9096-dc2b9b9b8989-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:20.469093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:20.469012 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:21.141909 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.141876 4981 generic.go:356] "Generic (PLEG): container finished" podID="63ed81d4-7c21-415f-b409-1aee30f3123e" containerID="5406fbaba17b54bae1e6deb8ea0abe696a1e06e269266d1761cabc571369b310" exitCode=0 Nov 01 22:37:21.142354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.141930 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" event={"ID":"63ed81d4-7c21-415f-b409-1aee30f3123e","Type":"ContainerDied","Data":"5406fbaba17b54bae1e6deb8ea0abe696a1e06e269266d1761cabc571369b310"} Nov 01 22:37:21.143684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.143662 4981 generic.go:356] "Generic (PLEG): container finished" podID="32810abe-3631-417c-89ba-da7a149a9abe" containerID="b2757e16babd7d4f01e96d6fcc47e28c98ecc3be3fe071230714ba945b6ff9bb" exitCode=0 Nov 01 22:37:21.143817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.143748 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" event={"ID":"32810abe-3631-417c-89ba-da7a149a9abe","Type":"ContainerDied","Data":"b2757e16babd7d4f01e96d6fcc47e28c98ecc3be3fe071230714ba945b6ff9bb"} Nov 01 22:37:21.143817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.143799 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" event={"ID":"32810abe-3631-417c-89ba-da7a149a9abe","Type":"ContainerStarted","Data":"0004935ff0431b0a9596a209c1341c14cc6844bf193d02a9cde11879eae404b4"} Nov 01 22:37:21.145183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.145161 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" Nov 01 22:37:21.145274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.145200 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9" event={"ID":"dc47d047-7b84-49b0-9096-dc2b9b9b8989","Type":"ContainerDied","Data":"cf517e96d9169860d233c0c5b581dce6116a693cd243dd410f77be69e48d5482"} Nov 01 22:37:21.145274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.145248 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf517e96d9169860d233c0c5b581dce6116a693cd243dd410f77be69e48d5482" Nov 01 22:37:21.146977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.146956 4981 generic.go:356] "Generic (PLEG): container finished" podID="220066e3-6f27-44c8-bd95-843708b3095e" containerID="c2313e74d0c10b68e34a2c8d7407debb0097c30e988da22fbdbbf2b1c1c65f8c" exitCode=0 Nov 01 22:37:21.147141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.147102 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" event={"ID":"220066e3-6f27-44c8-bd95-843708b3095e","Type":"ContainerDied","Data":"c2313e74d0c10b68e34a2c8d7407debb0097c30e988da22fbdbbf2b1c1c65f8c"} Nov 01 22:37:21.156700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.156678 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:21.522593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.522554 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:21.622424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.622390 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:21.864729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:21.864654 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:22.324347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.324320 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:22.392480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.392455 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:22.395722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.395704 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:22.440088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440056 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-xks-ca\") pod \"63ed81d4-7c21-415f-b409-1aee30f3123e\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " Nov 01 22:37:22.440238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440140 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-xks-ca\") pod \"220066e3-6f27-44c8-bd95-843708b3095e\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " Nov 01 22:37:22.440303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440228 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-xks-ca\") pod \"32810abe-3631-417c-89ba-da7a149a9abe\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " Nov 01 22:37:22.440373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440329 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-openshift-ca\") pod \"32810abe-3631-417c-89ba-da7a149a9abe\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " Nov 01 22:37:22.440426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440397 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ed81d4-7c21-415f-b409-1aee30f3123e-kube-api-access-ttzc4\" (UniqueName: \"kubernetes.io/projected/63ed81d4-7c21-415f-b409-1aee30f3123e-kube-api-access-ttzc4\") pod \"63ed81d4-7c21-415f-b409-1aee30f3123e\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " Nov 01 22:37:22.440493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440434 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220066e3-6f27-44c8-bd95-843708b3095e-kube-api-access-rsjfm\" (UniqueName: \"kubernetes.io/projected/220066e3-6f27-44c8-bd95-843708b3095e-kube-api-access-rsjfm\") pod \"220066e3-6f27-44c8-bd95-843708b3095e\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " Nov 01 22:37:22.440546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440496 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32810abe-3631-417c-89ba-da7a149a9abe-kube-api-access-cxngt\" (UniqueName: \"kubernetes.io/projected/32810abe-3631-417c-89ba-da7a149a9abe-kube-api-access-cxngt\") pod \"32810abe-3631-417c-89ba-da7a149a9abe\" (UID: \"32810abe-3631-417c-89ba-da7a149a9abe\") " Nov 01 22:37:22.440625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440529 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-openshift-ca\") pod \"63ed81d4-7c21-415f-b409-1aee30f3123e\" (UID: \"63ed81d4-7c21-415f-b409-1aee30f3123e\") " Nov 01 22:37:22.440686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440662 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-openshift-ca\") pod \"220066e3-6f27-44c8-bd95-843708b3095e\" (UID: \"220066e3-6f27-44c8-bd95-843708b3095e\") " Nov 01 22:37:22.441032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.440998 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-openshift-ca" pod "32810abe-3631-417c-89ba-da7a149a9abe" (UID: "32810abe-3631-417c-89ba-da7a149a9abe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:22.441614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.441559 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-openshift-ca" pod "220066e3-6f27-44c8-bd95-843708b3095e" (UID: "220066e3-6f27-44c8-bd95-843708b3095e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:22.441753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.441732 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.441832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.441761 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.441832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.441776 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-openshift-ca" pod "63ed81d4-7c21-415f-b409-1aee30f3123e" (UID: "63ed81d4-7c21-415f-b409-1aee30f3123e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:22.443085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.443060 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-xks-ca" pod "32810abe-3631-417c-89ba-da7a149a9abe" (UID: "32810abe-3631-417c-89ba-da7a149a9abe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:22.443190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.443064 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-xks-ca" pod "220066e3-6f27-44c8-bd95-843708b3095e" (UID: "220066e3-6f27-44c8-bd95-843708b3095e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:22.443190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.443064 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-xks-ca" pod "63ed81d4-7c21-415f-b409-1aee30f3123e" (UID: "63ed81d4-7c21-415f-b409-1aee30f3123e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:22.443486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.443459 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ed81d4-7c21-415f-b409-1aee30f3123e-kube-api-access-ttzc4" pod "63ed81d4-7c21-415f-b409-1aee30f3123e" (UID: "63ed81d4-7c21-415f-b409-1aee30f3123e"). InnerVolumeSpecName "kube-api-access-ttzc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:22.443881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.443860 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220066e3-6f27-44c8-bd95-843708b3095e-kube-api-access-rsjfm" pod "220066e3-6f27-44c8-bd95-843708b3095e" (UID: "220066e3-6f27-44c8-bd95-843708b3095e"). InnerVolumeSpecName "kube-api-access-rsjfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:22.444067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.444044 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32810abe-3631-417c-89ba-da7a149a9abe-kube-api-access-cxngt" pod "32810abe-3631-417c-89ba-da7a149a9abe" (UID: "32810abe-3631-417c-89ba-da7a149a9abe"). InnerVolumeSpecName "kube-api-access-cxngt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:22.543565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.543536 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220066e3-6f27-44c8-bd95-843708b3095e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.543565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.543564 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32810abe-3631-417c-89ba-da7a149a9abe-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.543763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.543600 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzc4\" (UniqueName: \"kubernetes.io/projected/63ed81d4-7c21-415f-b409-1aee30f3123e-kube-api-access-ttzc4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.543763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.543622 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsjfm\" (UniqueName: \"kubernetes.io/projected/220066e3-6f27-44c8-bd95-843708b3095e-kube-api-access-rsjfm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.543763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.543633 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxngt\" (UniqueName: \"kubernetes.io/projected/32810abe-3631-417c-89ba-da7a149a9abe-kube-api-access-cxngt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.543763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.543671 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.543763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.543681 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63ed81d4-7c21-415f-b409-1aee30f3123e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:22.548904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.548869 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:22.766539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.766511 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr"] Nov 01 22:37:22.767033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767019 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc47d047-7b84-49b0-9096-dc2b9b9b8989" containerName="revoke-keys" Nov 01 22:37:22.767074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767034 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc47d047-7b84-49b0-9096-dc2b9b9b8989" containerName="revoke-keys" Nov 01 22:37:22.767074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767047 4981 cpu_manager.go:414] "containerMap: removing container" podUID="63ed81d4-7c21-415f-b409-1aee30f3123e" containerName="revoke-keys" Nov 01 22:37:22.767074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767052 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ed81d4-7c21-415f-b409-1aee30f3123e" containerName="revoke-keys" Nov 01 22:37:22.767074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767073 4981 cpu_manager.go:414] "containerMap: removing container" podUID="220066e3-6f27-44c8-bd95-843708b3095e" containerName="revoke-keys" Nov 01 22:37:22.767192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767079 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220066e3-6f27-44c8-bd95-843708b3095e" containerName="revoke-keys" Nov 01 22:37:22.767192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767093 4981 cpu_manager.go:414] "containerMap: removing container" podUID="32810abe-3631-417c-89ba-da7a149a9abe" containerName="revoke-keys" Nov 01 22:37:22.767192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767098 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32810abe-3631-417c-89ba-da7a149a9abe" containerName="revoke-keys" Nov 01 22:37:22.767192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767184 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc47d047-7b84-49b0-9096-dc2b9b9b8989" containerName="revoke-keys" Nov 01 22:37:22.767319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767193 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="32810abe-3631-417c-89ba-da7a149a9abe" containerName="revoke-keys" Nov 01 22:37:22.767319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767200 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ed81d4-7c21-415f-b409-1aee30f3123e" containerName="revoke-keys" Nov 01 22:37:22.767319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.767210 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="220066e3-6f27-44c8-bd95-843708b3095e" containerName="revoke-keys" Nov 01 22:37:22.771534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.771519 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.781634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.781609 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e26733-openshift-default-dockercfg-794k8\"" Nov 01 22:37:22.786507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.786458 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr"] Nov 01 22:37:22.847346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847313 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4837ce-9330-4371-8e95-1dccb1851627-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847347 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847444 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847482 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4837ce-9330-4371-8e95-1dccb1851627-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847604 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847629 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2bw\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-kube-api-access-6g2bw\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847657 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847732 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.847840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.847825 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949224 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949269 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4837ce-9330-4371-8e95-1dccb1851627-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949295 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949349 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949375 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4837ce-9330-4371-8e95-1dccb1851627-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949448 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949481 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2bw\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-kube-api-access-6g2bw\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949523 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949608 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949703 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.949885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.949778 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.950075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.950053 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.950368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.950346 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4837ce-9330-4371-8e95-1dccb1851627-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.950470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.950451 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.951791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.951774 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4837ce-9330-4371-8e95-1dccb1851627-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.952176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.952158 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.957843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.957818 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2bw\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-kube-api-access-6g2bw\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:22.957995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:22.957970 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:23.087103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.087027 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:23.192872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.192826 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq"] Nov 01 22:37:23.199930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.199906 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" Nov 01 22:37:23.203510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.202020 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204073 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr" event={"ID":"63ed81d4-7c21-415f-b409-1aee30f3123e","Type":"ContainerDied","Data":"1d4e7bfe41404939e395ad771f09066808e27d9858fdcd65ffd59ffc7230c97c"} Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204117 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4e7bfe41404939e395ad771f09066808e27d9858fdcd65ffd59ffc7230c97c" Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204132 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7" event={"ID":"32810abe-3631-417c-89ba-da7a149a9abe","Type":"ContainerDied","Data":"0004935ff0431b0a9596a209c1341c14cc6844bf193d02a9cde11879eae404b4"} Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204154 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0004935ff0431b0a9596a209c1341c14cc6844bf193d02a9cde11879eae404b4" Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204170 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq"] Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204197 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" event={"ID":"220066e3-6f27-44c8-bd95-843708b3095e","Type":"ContainerDied","Data":"28d055d57d9381e7914c2a1422c7cc2e93513ba4a0e616cc21093f30863b0a6f"} Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204217 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d055d57d9381e7914c2a1422c7cc2e93513ba4a0e616cc21093f30863b0a6f" Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204285 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.205608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.204313 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b" Nov 01 22:37:23.267067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267034 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267069 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267097 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267240 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267301 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefeccec-8cb6-433e-99dc-195307e4621a-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267556 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267621 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tpg\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-kube-api-access-t2tpg\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267653 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.267812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.267672 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefeccec-8cb6-433e-99dc-195307e4621a-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.292604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.292555 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:23.371002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.370966 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371014 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371067 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371142 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371193 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefeccec-8cb6-433e-99dc-195307e4621a-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371327 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371355 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tpg\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-kube-api-access-t2tpg\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371391 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.371466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.371411 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefeccec-8cb6-433e-99dc-195307e4621a-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.372643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.372542 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefeccec-8cb6-433e-99dc-195307e4621a-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.372933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.372880 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.373413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.373368 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.373771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.373668 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.373897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.373885 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.375062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.375037 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr"] Nov 01 22:37:23.376271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.376225 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.376898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.376698 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefeccec-8cb6-433e-99dc-195307e4621a-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.379056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.379020 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:23.379179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.379104 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:23.379179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.379147 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:23.382191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.382172 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tpg\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-kube-api-access-t2tpg\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.388185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.388162 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.521488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.521455 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:23.814978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.814943 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq"] Nov 01 22:37:23.815278 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:23.815024 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefeccec_8cb6_433e_99dc_195307e4621a.slice/crio-0557fb24b7a8153be5f506f00f0fd849ce5b270ee40fbc16e2b9cfe378b381d0 WatchSource:0}: Error finding container 0557fb24b7a8153be5f506f00f0fd849ce5b270ee40fbc16e2b9cfe378b381d0: Status 404 returned error can't find the container with id 0557fb24b7a8153be5f506f00f0fd849ce5b270ee40fbc16e2b9cfe378b381d0 Nov 01 22:37:23.817466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.817428 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:23.817607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.817545 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:23.817685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.817609 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:23.901472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:23.901435 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:24.216642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.216603 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" event={"ID":"fefeccec-8cb6-433e-99dc-195307e4621a","Type":"ContainerStarted","Data":"a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed"} Nov 01 22:37:24.216823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.216673 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" event={"ID":"fefeccec-8cb6-433e-99dc-195307e4621a","Type":"ContainerStarted","Data":"0557fb24b7a8153be5f506f00f0fd849ce5b270ee40fbc16e2b9cfe378b381d0"} Nov 01 22:37:24.218326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.218301 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" event={"ID":"6f4837ce-9330-4371-8e95-1dccb1851627","Type":"ContainerStarted","Data":"ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22"} Nov 01 22:37:24.218474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.218344 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" event={"ID":"6f4837ce-9330-4371-8e95-1dccb1851627","Type":"ContainerStarted","Data":"b44168364c6f795ab1df77de582b32ffa91982621c8da4491dc854c79d8874af"} Nov 01 22:37:24.245822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.245774 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podStartSLOduration=1.245756633 podStartE2EDuration="1.245756633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:24.240296596 +0000 UTC m=+809.668174523" watchObservedRunningTime="2026-11-01 22:37:24.245756633 +0000 UTC m=+809.673634584" Nov 01 22:37:24.271590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.271537 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podStartSLOduration=2.271524438 podStartE2EDuration="2.271524438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:24.268021108 +0000 UTC m=+809.695899039" watchObservedRunningTime="2026-11-01 22:37:24.271524438 +0000 UTC m=+809.699402368" Nov 01 22:37:24.372256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.372214 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:24.525856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.525741 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:24.528984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.528944 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:24.529214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:24.529179 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:25.094751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:25.088746 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:25.094751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:25.093806 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:25.094751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:25.093862 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:25.420655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:25.420028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:25.523446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:25.522995 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:25.523446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:25.523056 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:26.088600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:26.087978 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.088600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:26.088037 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:26.531038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:26.531000 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:26.531677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:26.531631 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:26.856632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:26.856532 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:27.103640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:27.096746 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.103640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:27.096846 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:27.526042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:27.526002 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:27.526367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:27.526325 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:28.092112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.092077 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.092732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.092683 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:28.121721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.121675 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.159603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.155657 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.291602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.290823 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.319600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.318680 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:37:28.399658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.398693 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.457720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.457670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:37:28.496607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.495719 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:28.522027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.521994 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:28.522297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:28.522253 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:29.088007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.087966 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:29.088262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.088233 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:29.334637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.291671 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.334637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.292466 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:37:29.334637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.318829 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.334637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.319439 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:37:29.337640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.337240 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.400159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.399456 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.400159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.400117 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:37:29.465655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.460673 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.465655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.461341 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:37:29.523788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.523741 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:29.587601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.526692 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:37:29.587601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.528206 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:29.587601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.528258 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:29.751807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:29.751758 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:30.091654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.077675 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.118016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.117681 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gr8cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:37:30.118016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.117744 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.173628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.150684 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.173628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.150759 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:30.173628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.162227 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:37:30.249599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.248346 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.349654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.324003 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.349654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.325974 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.439627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.406437 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.539763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.539702 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:37:30.547731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.547686 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.568601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.565673 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:37:30.568601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.565739 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:30.615646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.590738 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:37:30.616508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.616478 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:30.616785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:30.616757 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:31.091096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.091051 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.227110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.226748 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:31.227110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.226801 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:31.229644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.227722 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:37:31.231629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.177559 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.232755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.231903 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:37:31.256142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.256100 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:37:31.258713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.258690 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:37:31.283623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.283562 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.335632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.327757 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.335632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.329378 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:37:31.442679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.440986 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:37:31.505190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.505159 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:31.547867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.547832 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:31.548106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.548076 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:31.548629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.548600 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.604605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.583656 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx"] Nov 01 22:37:31.654652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.641637 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:31.834651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.831813 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx"] Nov 01 22:37:31.834651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.831965 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:31.838612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.837915 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e5f47e\"/\"distinct-w0-e5f47e-kserve-self-signed-certs\"" Nov 01 22:37:31.980260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.979767 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-model-cache\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:31.980260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.979840 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-dshm\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:31.980260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.979877 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff04bd24-bb61-462b-acf7-1b11637a0eff-tls-certs\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:31.980260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.979952 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94rv\" (UniqueName: \"kubernetes.io/projected/ff04bd24-bb61-462b-acf7-1b11637a0eff-kube-api-access-d94rv\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:31.980260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.980017 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-home\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:31.980260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:31.980054 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-tmp-dir\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.089967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.089862 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-home\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.090613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.090196 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-tmp-dir\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.090613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.090310 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-model-cache\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.090613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.090365 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-dshm\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.090613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.090397 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff04bd24-bb61-462b-acf7-1b11637a0eff-tls-certs\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.090613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.090478 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94rv\" (UniqueName: \"kubernetes.io/projected/ff04bd24-bb61-462b-acf7-1b11637a0eff-kube-api-access-d94rv\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.091650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.091630 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-home\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.092047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.092027 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-tmp-dir\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.092406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.092385 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-model-cache\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.101569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.100680 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.101569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.100736 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 22:37:32.105174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.105131 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-dshm\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.145603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.125210 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94rv\" (UniqueName: \"kubernetes.io/projected/ff04bd24-bb61-462b-acf7-1b11637a0eff-kube-api-access-d94rv\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.145603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.136231 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff04bd24-bb61-462b-acf7-1b11637a0eff-tls-certs\") pod \"distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.239318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.234658 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:37:32.265511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.246748 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.265511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.264665 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.265511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.265180 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:32.375950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.360058 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.375950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.361431 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:32.379247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.379173 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:37:32.396712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.396436 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:37:32.425146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.424102 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:37:32.425146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.424965 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:37:32.425146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.425040 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:37:32.440621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.439992 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:37:32.441613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.441175 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:37:32.450968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.450863 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:37:32.518330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.518234 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:32.522177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.522148 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 01 22:37:32.522300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.522202 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 01 22:37:32.789721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.789687 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx"] Nov 01 22:37:32.791099 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:32.791070 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff04bd24_bb61_462b_acf7_1b11637a0eff.slice/crio-30a58db206c30125822def58472d757ad8b704730098979cfb7a2a29b086d6b9 WatchSource:0}: Error finding container 30a58db206c30125822def58472d757ad8b704730098979cfb7a2a29b086d6b9: Status 404 returned error can't find the container with id 30a58db206c30125822def58472d757ad8b704730098979cfb7a2a29b086d6b9 Nov 01 22:37:32.923349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.923311 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" event={"ID":"ff04bd24-bb61-462b-acf7-1b11637a0eff","Type":"ContainerStarted","Data":"2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc"} Nov 01 22:37:32.923534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.923361 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" event={"ID":"ff04bd24-bb61-462b-acf7-1b11637a0eff","Type":"ContainerStarted","Data":"30a58db206c30125822def58472d757ad8b704730098979cfb7a2a29b086d6b9"} Nov 01 22:37:32.946242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:32.946191 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podStartSLOduration=1.94617193 podStartE2EDuration="1.94617193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:32.940094601 +0000 UTC m=+818.367972534" watchObservedRunningTime="2026-11-01 22:37:32.94617193 +0000 UTC m=+818.374049865" Nov 01 22:37:33.138563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.138477 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6"] Nov 01 22:37:33.142711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.142685 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.151995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.151965 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6"] Nov 01 22:37:33.155313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.152321 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:37:33.155313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.152992 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:33.155313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.153063 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:33.155313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.153114 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:33.189245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.189187 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.189413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.189343 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.189413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.189366 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frszk\" (UniqueName: \"kubernetes.io/projected/f8a81a22-143f-4faa-8ca5-b7d152c74c85-kube-api-access-frszk\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.266619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.266563 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:33.290637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.290524 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.290808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.290658 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frszk\" (UniqueName: \"kubernetes.io/projected/f8a81a22-143f-4faa-8ca5-b7d152c74c85-kube-api-access-frszk\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.290877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.290835 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.291308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.291282 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.293546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.293517 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.318774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.318744 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frszk\" (UniqueName: \"kubernetes.io/projected/f8a81a22-143f-4faa-8ca5-b7d152c74c85-kube-api-access-frszk\") pod \"maas-api-revoke-sub-4aa639cd-g5gg6\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.473245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.473210 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:33.494650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.494613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:33.760219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.760186 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6"] Nov 01 22:37:33.760474 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:33.760451 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a81a22_143f_4faa_8ca5_b7d152c74c85.slice/crio-3b73880d1b39456f7b41672025e703a9da3bc2417ea2b1d835afa27e279599cc WatchSource:0}: Error finding container 3b73880d1b39456f7b41672025e703a9da3bc2417ea2b1d835afa27e279599cc: Status 404 returned error can't find the container with id 3b73880d1b39456f7b41672025e703a9da3bc2417ea2b1d835afa27e279599cc Nov 01 22:37:33.937812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:33.937729 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" event={"ID":"f8a81a22-143f-4faa-8ca5-b7d152c74c85","Type":"ContainerStarted","Data":"3b73880d1b39456f7b41672025e703a9da3bc2417ea2b1d835afa27e279599cc"} Nov 01 22:37:34.103288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.103251 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:37:34.103459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.103314 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:37:34.211371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.211342 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:34.211918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.211890 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:34.212750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.212729 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 22:37:34.213792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.213767 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:34.214045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.214029 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:34.240791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.240764 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 22:37:34.255521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.255498 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:34.898664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.898590 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g"] Nov 01 22:37:34.902800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.902782 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:34.912714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.912690 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:34.912812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.912798 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:34.912905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.912891 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvv5\" (UniqueName: \"kubernetes.io/projected/1e3d7c49-e789-40da-afc8-72f69100c0f6-kube-api-access-ztvv5\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:34.945235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.945210 4981 generic.go:356] "Generic (PLEG): container finished" podID="f8a81a22-143f-4faa-8ca5-b7d152c74c85" containerID="3c980dca3f7b462471dbcc437685b0d4782316272f68b9438aa38a697df5e1a5" exitCode=0 Nov 01 22:37:34.945336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.945286 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" event={"ID":"f8a81a22-143f-4faa-8ca5-b7d152c74c85","Type":"ContainerDied","Data":"3c980dca3f7b462471dbcc437685b0d4782316272f68b9438aa38a697df5e1a5"} Nov 01 22:37:34.997518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:34.997489 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g"] Nov 01 22:37:35.014213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.014177 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:35.014367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.014329 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvv5\" (UniqueName: \"kubernetes.io/projected/1e3d7c49-e789-40da-afc8-72f69100c0f6-kube-api-access-ztvv5\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:35.014661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.014641 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:35.015502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.015471 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:35.016870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.016842 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:35.034756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.034704 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvv5\" (UniqueName: \"kubernetes.io/projected/1e3d7c49-e789-40da-afc8-72f69100c0f6-kube-api-access-ztvv5\") pod \"maas-api-revoke-sub-2f5fe5fe-hdk8g\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:35.216562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.216533 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:35.428455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.428415 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:35.505264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.505237 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g"] Nov 01 22:37:35.505389 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:35.505305 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3d7c49_e789_40da_afc8_72f69100c0f6.slice/crio-b98d58657df8ac6f0d955f1ecf8b978dd7b2f5852a516e0e1cfcf1f8b88f1ddb WatchSource:0}: Error finding container b98d58657df8ac6f0d955f1ecf8b978dd7b2f5852a516e0e1cfcf1f8b88f1ddb: Status 404 returned error can't find the container with id b98d58657df8ac6f0d955f1ecf8b978dd7b2f5852a516e0e1cfcf1f8b88f1ddb Nov 01 22:37:35.953343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.953250 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" event={"ID":"1e3d7c49-e789-40da-afc8-72f69100c0f6","Type":"ContainerStarted","Data":"7c93948d0717eb357dfb3d0b39a87cf8cb45261779fb13c7b0136fce91d4d8be"} Nov 01 22:37:35.953343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.953335 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" event={"ID":"1e3d7c49-e789-40da-afc8-72f69100c0f6","Type":"ContainerStarted","Data":"b98d58657df8ac6f0d955f1ecf8b978dd7b2f5852a516e0e1cfcf1f8b88f1ddb"} Nov 01 22:37:35.972413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:35.972369 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" podStartSLOduration=1.9723583310000001 podStartE2EDuration="1.972358331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:35.970891076 +0000 UTC m=+821.398769007" watchObservedRunningTime="2026-11-01 22:37:35.972358331 +0000 UTC m=+821.400236262" Nov 01 22:37:36.093316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.093291 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:36.129321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.129285 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-xks-ca\") pod \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " Nov 01 22:37:36.129684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.129659 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-openshift-ca\") pod \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " Nov 01 22:37:36.129791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.129758 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a81a22-143f-4faa-8ca5-b7d152c74c85-kube-api-access-frszk\" (UniqueName: \"kubernetes.io/projected/f8a81a22-143f-4faa-8ca5-b7d152c74c85-kube-api-access-frszk\") pod \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\" (UID: \"f8a81a22-143f-4faa-8ca5-b7d152c74c85\") " Nov 01 22:37:36.130171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.130139 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-openshift-ca" pod "f8a81a22-143f-4faa-8ca5-b7d152c74c85" (UID: "f8a81a22-143f-4faa-8ca5-b7d152c74c85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:36.131712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.131683 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-xks-ca" pod "f8a81a22-143f-4faa-8ca5-b7d152c74c85" (UID: "f8a81a22-143f-4faa-8ca5-b7d152c74c85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:36.132118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.132095 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a81a22-143f-4faa-8ca5-b7d152c74c85-kube-api-access-frszk" pod "f8a81a22-143f-4faa-8ca5-b7d152c74c85" (UID: "f8a81a22-143f-4faa-8ca5-b7d152c74c85"). InnerVolumeSpecName "kube-api-access-frszk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:36.156737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.156712 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:36.233018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.232919 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:36.233018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.232961 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frszk\" (UniqueName: \"kubernetes.io/projected/f8a81a22-143f-4faa-8ca5-b7d152c74c85-kube-api-access-frszk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:36.233018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.232977 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8a81a22-143f-4faa-8ca5-b7d152c74c85-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:36.865257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.865227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:36.961669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.961634 4981 generic.go:356] "Generic (PLEG): container finished" podID="1e3d7c49-e789-40da-afc8-72f69100c0f6" containerID="7c93948d0717eb357dfb3d0b39a87cf8cb45261779fb13c7b0136fce91d4d8be" exitCode=0 Nov 01 22:37:36.962096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.961721 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" event={"ID":"1e3d7c49-e789-40da-afc8-72f69100c0f6","Type":"ContainerDied","Data":"7c93948d0717eb357dfb3d0b39a87cf8cb45261779fb13c7b0136fce91d4d8be"} Nov 01 22:37:36.963149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.963123 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" event={"ID":"f8a81a22-143f-4faa-8ca5-b7d152c74c85","Type":"ContainerDied","Data":"3b73880d1b39456f7b41672025e703a9da3bc2417ea2b1d835afa27e279599cc"} Nov 01 22:37:36.963149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.963145 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6" Nov 01 22:37:36.963291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:36.963162 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b73880d1b39456f7b41672025e703a9da3bc2417ea2b1d835afa27e279599cc" Nov 01 22:37:37.517480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:37.517452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:38.119370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.119343 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:38.157963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.157926 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-xks-ca\") pod \"1e3d7c49-e789-40da-afc8-72f69100c0f6\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " Nov 01 22:37:38.158114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.158055 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3d7c49-e789-40da-afc8-72f69100c0f6-kube-api-access-ztvv5\" (UniqueName: \"kubernetes.io/projected/1e3d7c49-e789-40da-afc8-72f69100c0f6-kube-api-access-ztvv5\") pod \"1e3d7c49-e789-40da-afc8-72f69100c0f6\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " Nov 01 22:37:38.158187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.158171 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-openshift-ca\") pod \"1e3d7c49-e789-40da-afc8-72f69100c0f6\" (UID: \"1e3d7c49-e789-40da-afc8-72f69100c0f6\") " Nov 01 22:37:38.158531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.158497 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-openshift-ca" pod "1e3d7c49-e789-40da-afc8-72f69100c0f6" (UID: "1e3d7c49-e789-40da-afc8-72f69100c0f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:38.158778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.158753 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:38.160186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.160157 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-xks-ca" pod "1e3d7c49-e789-40da-afc8-72f69100c0f6" (UID: "1e3d7c49-e789-40da-afc8-72f69100c0f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:38.160286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.160239 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3d7c49-e789-40da-afc8-72f69100c0f6-kube-api-access-ztvv5" pod "1e3d7c49-e789-40da-afc8-72f69100c0f6" (UID: "1e3d7c49-e789-40da-afc8-72f69100c0f6"). InnerVolumeSpecName "kube-api-access-ztvv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:38.259380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.259301 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:38.259688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.259665 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e3d7c49-e789-40da-afc8-72f69100c0f6-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:38.259807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.259692 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvv5\" (UniqueName: \"kubernetes.io/projected/1e3d7c49-e789-40da-afc8-72f69100c0f6-kube-api-access-ztvv5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:38.882074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.882041 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:38.979363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.979329 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" Nov 01 22:37:38.979363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.979336 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g" event={"ID":"1e3d7c49-e789-40da-afc8-72f69100c0f6","Type":"ContainerDied","Data":"b98d58657df8ac6f0d955f1ecf8b978dd7b2f5852a516e0e1cfcf1f8b88f1ddb"} Nov 01 22:37:38.979639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:38.979490 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b98d58657df8ac6f0d955f1ecf8b978dd7b2f5852a516e0e1cfcf1f8b88f1ddb" Nov 01 22:37:39.631595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:39.631543 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:40.329849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:40.329813 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:41.003093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:41.003059 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:41.774421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:41.774389 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:42.266784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:42.266748 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:42.279195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:42.279173 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:42.706358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:42.706327 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:43.016125 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.016037 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:37:43.366641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.366556 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g"] Nov 01 22:37:43.367132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.367118 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1e3d7c49-e789-40da-afc8-72f69100c0f6" containerName="revoke-keys" Nov 01 22:37:43.367175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.367133 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3d7c49-e789-40da-afc8-72f69100c0f6" containerName="revoke-keys" Nov 01 22:37:43.367175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.367157 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f8a81a22-143f-4faa-8ca5-b7d152c74c85" containerName="revoke-keys" Nov 01 22:37:43.367175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.367162 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a81a22-143f-4faa-8ca5-b7d152c74c85" containerName="revoke-keys" Nov 01 22:37:43.367266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.367255 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3d7c49-e789-40da-afc8-72f69100c0f6" containerName="revoke-keys" Nov 01 22:37:43.367303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.367270 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a81a22-143f-4faa-8ca5-b7d152c74c85" containerName="revoke-keys" Nov 01 22:37:43.370771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.370755 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.375044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.375019 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:43.375913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.375869 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:43.375913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.375872 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:43.376470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.376451 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:37:43.379914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.379832 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g"] Nov 01 22:37:43.420831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.420803 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.421000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.420846 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4np\" (UniqueName: \"kubernetes.io/projected/44e6ea18-1cdb-4675-af33-90af7b44f788-kube-api-access-vs4np\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.421000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.420981 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.522425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.522393 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.522634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.522440 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4np\" (UniqueName: \"kubernetes.io/projected/44e6ea18-1cdb-4675-af33-90af7b44f788-kube-api-access-vs4np\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.522634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.522484 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.523223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.523192 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.524807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.524777 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.538883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.538858 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l"] Nov 01 22:37:43.543305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.543284 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.548128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.548108 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4np\" (UniqueName: \"kubernetes.io/projected/44e6ea18-1cdb-4675-af33-90af7b44f788-kube-api-access-vs4np\") pod \"maas-api-revoke-sub-81701ab8-f7n5g\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.548382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.548359 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"distinct-w4-2f3705-kserve-self-signed-certs\"" Nov 01 22:37:43.580614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.580589 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l"] Nov 01 22:37:43.623366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.623287 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjt9k\" (UniqueName: \"kubernetes.io/projected/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-kube-api-access-sjt9k\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.623366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.623341 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tmp-dir\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.623535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.623491 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-model-cache\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.623570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.623560 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tls-certs\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.623680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.623665 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-home\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.623776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.623761 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-dshm\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.685641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.685606 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:43.687778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.687509 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:43.725549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.725515 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tls-certs\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.725719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.725590 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-home\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.725719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.725686 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-dshm\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.725719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.725715 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjt9k\" (UniqueName: \"kubernetes.io/projected/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-kube-api-access-sjt9k\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.725891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.725771 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tmp-dir\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.725891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.725881 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-model-cache\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.726112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.726052 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-home\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.726251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.726227 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-model-cache\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.726322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.726283 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tmp-dir\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.728116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.728094 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-dshm\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.728260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.728231 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tls-certs\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.738228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.738191 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjt9k\" (UniqueName: \"kubernetes.io/projected/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-kube-api-access-sjt9k\") pod \"distinct-w4-2f3705-kserve-5d5548fb76-p4z5l\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.867511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.867474 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:43.885673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.885650 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:43.988954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:43.988925 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g"] Nov 01 22:37:43.989201 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:43.989178 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e6ea18_1cdb_4675_af33_90af7b44f788.slice/crio-8448fd27307bab2368b503a76ef585b5a184d569479cc4ea258c81bb6154b0ff WatchSource:0}: Error finding container 8448fd27307bab2368b503a76ef585b5a184d569479cc4ea258c81bb6154b0ff: Status 404 returned error can't find the container with id 8448fd27307bab2368b503a76ef585b5a184d569479cc4ea258c81bb6154b0ff Nov 01 22:37:44.012811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:44.012783 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" event={"ID":"44e6ea18-1cdb-4675-af33-90af7b44f788","Type":"ContainerStarted","Data":"8448fd27307bab2368b503a76ef585b5a184d569479cc4ea258c81bb6154b0ff"} Nov 01 22:37:44.149202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:44.149175 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l"] Nov 01 22:37:44.152059 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:44.152027 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3b59a77_aff1_4b88_a94b_0bf68219f2a5.slice/crio-39900e65b35eaf3b2579d9bc867fee612a70c965c425033aa3638070e3d7e1d2 WatchSource:0}: Error finding container 39900e65b35eaf3b2579d9bc867fee612a70c965c425033aa3638070e3d7e1d2: Status 404 returned error can't find the container with id 39900e65b35eaf3b2579d9bc867fee612a70c965c425033aa3638070e3d7e1d2 Nov 01 22:37:44.524969 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:37:44.524932 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:37:44.556172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:44.556140 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:45.020784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.020753 4981 generic.go:356] "Generic (PLEG): container finished" podID="44e6ea18-1cdb-4675-af33-90af7b44f788" containerID="dcf2d08e0955cfb2ae40d4a4d402363f05e00c458a8d7ceb8e1a1b0eb7c484d8" exitCode=0 Nov 01 22:37:45.020955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.020831 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" event={"ID":"44e6ea18-1cdb-4675-af33-90af7b44f788","Type":"ContainerDied","Data":"dcf2d08e0955cfb2ae40d4a4d402363f05e00c458a8d7ceb8e1a1b0eb7c484d8"} Nov 01 22:37:45.022518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.022491 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" event={"ID":"d3b59a77-aff1-4b88-a94b-0bf68219f2a5","Type":"ContainerStarted","Data":"1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c"} Nov 01 22:37:45.022650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.022534 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" event={"ID":"d3b59a77-aff1-4b88-a94b-0bf68219f2a5","Type":"ContainerStarted","Data":"39900e65b35eaf3b2579d9bc867fee612a70c965c425033aa3638070e3d7e1d2"} Nov 01 22:37:45.061467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.061382 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podStartSLOduration=2.061369024 podStartE2EDuration="2.061369024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:45.061275562 +0000 UTC m=+830.489153493" watchObservedRunningTime="2026-11-01 22:37:45.061369024 +0000 UTC m=+830.489246955" Nov 01 22:37:45.119472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.119439 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72"] Nov 01 22:37:45.124052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.124030 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.143154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.143122 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"openshift-service-ca.crt\"" Nov 01 22:37:45.146318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.146293 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"kube-root-ca.crt\"" Nov 01 22:37:45.146473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.146295 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"default-dockercfg-p5mnk\"" Nov 01 22:37:45.146569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.146515 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9ad303\"/\"distinct-w2-9ad303-kserve-self-signed-certs\"" Nov 01 22:37:45.149177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.149142 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72"] Nov 01 22:37:45.215411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.215379 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:45.246815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.246790 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-home\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.246954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.246842 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-model-cache\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.247000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.246976 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf93a3f3-247b-4328-8abf-6173fb4543ec-tls-certs\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.247111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.247096 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zmq\" (UniqueName: \"kubernetes.io/projected/bf93a3f3-247b-4328-8abf-6173fb4543ec-kube-api-access-p5zmq\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.247157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.247145 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-dshm\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.247193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.247166 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-tmp-dir\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.352647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.351723 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zmq\" (UniqueName: \"kubernetes.io/projected/bf93a3f3-247b-4328-8abf-6173fb4543ec-kube-api-access-p5zmq\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.352647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.351808 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-dshm\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.352647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.351839 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-tmp-dir\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.352647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.351907 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-home\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.352647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.351969 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-model-cache\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.352647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.352055 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf93a3f3-247b-4328-8abf-6173fb4543ec-tls-certs\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.355603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.353092 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-home\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.356804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.356759 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-model-cache\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.356981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.356763 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-tmp-dir\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.362312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.362246 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zmq\" (UniqueName: \"kubernetes.io/projected/bf93a3f3-247b-4328-8abf-6173fb4543ec-kube-api-access-p5zmq\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.364865 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.364843 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-dshm\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.384199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.379479 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf93a3f3-247b-4328-8abf-6173fb4543ec-tls-certs\") pod \"distinct-w2-9ad303-kserve-585b6c6d77-8tp72\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.440073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.440041 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:45.647393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.647312 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:45.726744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.726714 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72"] Nov 01 22:37:45.729395 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:45.729224 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf93a3f3_247b_4328_8abf_6173fb4543ec.slice/crio-e83c443097b3781e0966c461c4b76dbc5e58129d7888c790a8d68472ea5dc107 WatchSource:0}: Error finding container e83c443097b3781e0966c461c4b76dbc5e58129d7888c790a8d68472ea5dc107: Status 404 returned error can't find the container with id e83c443097b3781e0966c461c4b76dbc5e58129d7888c790a8d68472ea5dc107 Nov 01 22:37:45.868041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:45.868010 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:46.039599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.039399 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" event={"ID":"bf93a3f3-247b-4328-8abf-6173fb4543ec","Type":"ContainerStarted","Data":"38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d"} Nov 01 22:37:46.039599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.039453 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" event={"ID":"bf93a3f3-247b-4328-8abf-6173fb4543ec","Type":"ContainerStarted","Data":"e83c443097b3781e0966c461c4b76dbc5e58129d7888c790a8d68472ea5dc107"} Nov 01 22:37:46.073797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.073713 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podStartSLOduration=1.073693009 podStartE2EDuration="1.073693009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:46.07297896 +0000 UTC m=+831.500856892" watchObservedRunningTime="2026-11-01 22:37:46.073693009 +0000 UTC m=+831.501570942" Nov 01 22:37:46.239337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.239300 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:46.365138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.365041 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e6ea18-1cdb-4675-af33-90af7b44f788-kube-api-access-vs4np\" (UniqueName: \"kubernetes.io/projected/44e6ea18-1cdb-4675-af33-90af7b44f788-kube-api-access-vs4np\") pod \"44e6ea18-1cdb-4675-af33-90af7b44f788\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " Nov 01 22:37:46.365318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.365141 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-openshift-ca\") pod \"44e6ea18-1cdb-4675-af33-90af7b44f788\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " Nov 01 22:37:46.365318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.365188 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-xks-ca\") pod \"44e6ea18-1cdb-4675-af33-90af7b44f788\" (UID: \"44e6ea18-1cdb-4675-af33-90af7b44f788\") " Nov 01 22:37:46.365813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.365758 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-openshift-ca" pod "44e6ea18-1cdb-4675-af33-90af7b44f788" (UID: "44e6ea18-1cdb-4675-af33-90af7b44f788"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:46.367944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.367915 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e6ea18-1cdb-4675-af33-90af7b44f788-kube-api-access-vs4np" pod "44e6ea18-1cdb-4675-af33-90af7b44f788" (UID: "44e6ea18-1cdb-4675-af33-90af7b44f788"). InnerVolumeSpecName "kube-api-access-vs4np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:46.367944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.367926 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-xks-ca" pod "44e6ea18-1cdb-4675-af33-90af7b44f788" (UID: "44e6ea18-1cdb-4675-af33-90af7b44f788"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:46.440961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.440899 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:46.467422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.467389 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4np\" (UniqueName: \"kubernetes.io/projected/44e6ea18-1cdb-4675-af33-90af7b44f788-kube-api-access-vs4np\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.467422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.467414 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.467422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.467424 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44e6ea18-1cdb-4675-af33-90af7b44f788-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:46.561839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:46.560478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:47.051316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:47.051287 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" Nov 01 22:37:47.051771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:47.051324 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g" event={"ID":"44e6ea18-1cdb-4675-af33-90af7b44f788","Type":"ContainerDied","Data":"8448fd27307bab2368b503a76ef585b5a184d569479cc4ea258c81bb6154b0ff"} Nov 01 22:37:47.051771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:47.051382 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8448fd27307bab2368b503a76ef585b5a184d569479cc4ea258c81bb6154b0ff" Nov 01 22:37:47.746248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:47.746219 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:48.661516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:48.661481 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:49.598354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:49.598317 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:50.268475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:50.268444 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:51.057795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:51.057766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:51.738876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:51.738833 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:52.364352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:52.364319 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:52.996085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:52.996052 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj"] Nov 01 22:37:52.996846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:52.996825 4981 cpu_manager.go:414] "containerMap: removing container" podUID="44e6ea18-1cdb-4675-af33-90af7b44f788" containerName="revoke-keys" Nov 01 22:37:52.996846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:52.996846 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e6ea18-1cdb-4675-af33-90af7b44f788" containerName="revoke-keys" Nov 01 22:37:52.997042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:52.997028 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e6ea18-1cdb-4675-af33-90af7b44f788" containerName="revoke-keys" Nov 01 22:37:53.001708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.001686 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.013831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.013804 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e5f47e-openshift-default-dockercfg-lfvrz\"" Nov 01 22:37:53.017327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.017299 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj"] Nov 01 22:37:53.051404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051352 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.051618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051491 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.051618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051550 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.051757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051687 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.051757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051721 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.051875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051843 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.051931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051874 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.051931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.051920 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4704702-163d-464d-b7fa-c0723a4ba01b-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.052035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.052021 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4z8\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-kube-api-access-vz4z8\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.152887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.152852 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153076 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.152895 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153076 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153021 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153076 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153066 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153129 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153160 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153186 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4704702-163d-464d-b7fa-c0723a4ba01b-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153389 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153273 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4z8\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-kube-api-access-vz4z8\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153389 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153282 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153395 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153461 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153780 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.153969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.153949 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4704702-163d-464d-b7fa-c0723a4ba01b-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.154050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.154027 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.154961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.154939 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:53.155695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.155671 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.155898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.155882 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.161864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.161840 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.162044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.162023 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4z8\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-kube-api-access-vz4z8\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.318288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.318202 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:53.607848 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:53.607813 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4704702_163d_464d_b7fa_c0723a4ba01b.slice/crio-fc32040fbc8c8a5e7e77aef52c73b018d495328d5531c5fba492a8ba4a2c6732 WatchSource:0}: Error finding container fc32040fbc8c8a5e7e77aef52c73b018d495328d5531c5fba492a8ba4a2c6732: Status 404 returned error can't find the container with id fc32040fbc8c8a5e7e77aef52c73b018d495328d5531c5fba492a8ba4a2c6732 Nov 01 22:37:53.607848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.607835 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj"] Nov 01 22:37:53.610896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.610865 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:53.610983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.610928 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:53.610983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.610957 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:37:53.867684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.867600 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:53.868053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.868028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:53.880721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:53.880698 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:54.109781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.109739 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" event={"ID":"e4704702-163d-464d-b7fa-c0723a4ba01b","Type":"ContainerStarted","Data":"f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe"} Nov 01 22:37:54.110309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.109800 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" event={"ID":"e4704702-163d-464d-b7fa-c0723a4ba01b","Type":"ContainerStarted","Data":"fc32040fbc8c8a5e7e77aef52c73b018d495328d5531c5fba492a8ba4a2c6732"} Nov 01 22:37:54.127421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.127315 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:37:54.135606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.134415 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podStartSLOduration=2.134396814 podStartE2EDuration="2.134396814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:54.129782743 +0000 UTC m=+839.557660687" watchObservedRunningTime="2026-11-01 22:37:54.134396814 +0000 UTC m=+839.562274746" Nov 01 22:37:54.319264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.319226 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:37:54.320622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.320567 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:37:54.320750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.320652 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 22:37:54.558055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.558016 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:54.659550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:54.659510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:55.319192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.319157 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:37:55.319664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.319243 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 22:37:55.413503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.413460 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc"] Nov 01 22:37:55.418959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.418937 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.434536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.434514 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:37:55.434695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.434594 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:37:55.434767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.434705 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:37:55.436249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.436220 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:37:55.440817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.440797 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:55.453746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.453726 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:55.515599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.509805 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.515599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.509960 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.515599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.510006 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69bb9\" (UniqueName: \"kubernetes.io/projected/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-kube-api-access-69bb9\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.530797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.525448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:55.531349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.531319 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc"] Nov 01 22:37:55.611900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.611814 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.611900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.611871 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69bb9\" (UniqueName: \"kubernetes.io/projected/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-kube-api-access-69bb9\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.612131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.612067 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.612834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.612814 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.614367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.614337 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.628338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.628311 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69bb9\" (UniqueName: \"kubernetes.io/projected/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-kube-api-access-69bb9\") pod \"maas-api-revoke-sub-ed54806f-wlnhc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:55.733019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:55.732987 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:56.050878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:56.050845 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc"] Nov 01 22:37:56.051356 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:37:56.051334 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aad3b0c_d4de_46f6_aa85_50f35e4907cc.slice/crio-2db973c92e4b339fae4440d92c8596ee53df2ee9c253943ec889da8f8703f975 WatchSource:0}: Error finding container 2db973c92e4b339fae4440d92c8596ee53df2ee9c253943ec889da8f8703f975: Status 404 returned error can't find the container with id 2db973c92e4b339fae4440d92c8596ee53df2ee9c253943ec889da8f8703f975 Nov 01 22:37:56.125757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:56.125722 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" event={"ID":"8aad3b0c-d4de-46f6-aa85-50f35e4907cc","Type":"ContainerStarted","Data":"2db973c92e4b339fae4440d92c8596ee53df2ee9c253943ec889da8f8703f975"} Nov 01 22:37:56.136936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:56.136908 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:37:56.319119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:56.319035 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:37:56.319119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:56.319091 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 22:37:56.596213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:56.596077 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:57.135473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:57.135446 4981 generic.go:356] "Generic (PLEG): container finished" podID="8aad3b0c-d4de-46f6-aa85-50f35e4907cc" containerID="0672b9fd11f97ba2d29d699d96e6e37775919e5e5c1ebc5e18df80597987ff45" exitCode=0 Nov 01 22:37:57.135630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:57.135487 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" event={"ID":"8aad3b0c-d4de-46f6-aa85-50f35e4907cc","Type":"ContainerDied","Data":"0672b9fd11f97ba2d29d699d96e6e37775919e5e5c1ebc5e18df80597987ff45"} Nov 01 22:37:57.318921 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:57.318887 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:37:57.319093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:57.318951 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 22:37:57.372639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:57.372534 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:58.062836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.062800 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:58.319320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.319288 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:37:58.319454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.319344 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 22:37:58.334606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.334570 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:58.448626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.448565 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-kube-api-access-69bb9\" (UniqueName: \"kubernetes.io/projected/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-kube-api-access-69bb9\") pod \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " Nov 01 22:37:58.449012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.448657 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-xks-ca\") pod \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " Nov 01 22:37:58.449012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.448776 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-openshift-ca\") pod \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\" (UID: \"8aad3b0c-d4de-46f6-aa85-50f35e4907cc\") " Nov 01 22:37:58.449167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.449143 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-openshift-ca" pod "8aad3b0c-d4de-46f6-aa85-50f35e4907cc" (UID: "8aad3b0c-d4de-46f6-aa85-50f35e4907cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:37:58.449344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.449328 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:58.450736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.450706 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-xks-ca" pod "8aad3b0c-d4de-46f6-aa85-50f35e4907cc" (UID: "8aad3b0c-d4de-46f6-aa85-50f35e4907cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:37:58.450873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.450735 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-kube-api-access-69bb9" pod "8aad3b0c-d4de-46f6-aa85-50f35e4907cc" (UID: "8aad3b0c-d4de-46f6-aa85-50f35e4907cc"). InnerVolumeSpecName "kube-api-access-69bb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:37:58.550733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.550697 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69bb9\" (UniqueName: \"kubernetes.io/projected/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-kube-api-access-69bb9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:58.550733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.550727 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aad3b0c-d4de-46f6-aa85-50f35e4907cc-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:37:58.766728 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:58.766698 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:59.154338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.154244 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" event={"ID":"8aad3b0c-d4de-46f6-aa85-50f35e4907cc","Type":"ContainerDied","Data":"2db973c92e4b339fae4440d92c8596ee53df2ee9c253943ec889da8f8703f975"} Nov 01 22:37:59.154338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.154300 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db973c92e4b339fae4440d92c8596ee53df2ee9c253943ec889da8f8703f975" Nov 01 22:37:59.154338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.154250 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc" Nov 01 22:37:59.318844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.318808 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 22:37:59.319016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.318860 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 22:37:59.419600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.419501 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz"] Nov 01 22:37:59.420138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.420119 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8aad3b0c-d4de-46f6-aa85-50f35e4907cc" containerName="revoke-keys" Nov 01 22:37:59.420138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.420135 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aad3b0c-d4de-46f6-aa85-50f35e4907cc" containerName="revoke-keys" Nov 01 22:37:59.420311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.420223 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aad3b0c-d4de-46f6-aa85-50f35e4907cc" containerName="revoke-keys" Nov 01 22:37:59.424179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.424155 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.424340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.424312 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz"] Nov 01 22:37:59.426054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.426031 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:59.427561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.427543 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-927b68\"/\"distinct-2-w6-927b68-kserve-self-signed-certs\"" Nov 01 22:37:59.527336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.527303 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:37:59.564745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.564707 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7221b-ee32-4b06-854e-b90394314b45-tls-certs\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.565318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.565290 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-tmp-dir\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.565460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.565443 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-dshm\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.565674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.565552 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6d7\" (UniqueName: \"kubernetes.io/projected/42e7221b-ee32-4b06-854e-b90394314b45-kube-api-access-sr6d7\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.565674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.565665 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-model-cache\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.565828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.565747 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-home\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667026 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-home\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667090 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7221b-ee32-4b06-854e-b90394314b45-tls-certs\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667182 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-tmp-dir\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667202 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-dshm\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667218 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6d7\" (UniqueName: \"kubernetes.io/projected/42e7221b-ee32-4b06-854e-b90394314b45-kube-api-access-sr6d7\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667266 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-model-cache\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667486 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-home\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667882 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-tmp-dir\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.667994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.667909 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-model-cache\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.670031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.669974 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-dshm\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.670413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.670393 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7221b-ee32-4b06-854e-b90394314b45-tls-certs\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.676040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.676015 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6d7\" (UniqueName: \"kubernetes.io/projected/42e7221b-ee32-4b06-854e-b90394314b45-kube-api-access-sr6d7\") pod \"distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:37:59.738689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:37:59.738650 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:38:00.018081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.018053 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz"] Nov 01 22:38:00.020397 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:00.020372 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42e7221b_ee32_4b06_854e_b90394314b45.slice/crio-6618a76aef159a2066167066390470eb70c659262c8dff14c19d9ea0f8fe4f61 WatchSource:0}: Error finding container 6618a76aef159a2066167066390470eb70c659262c8dff14c19d9ea0f8fe4f61: Status 404 returned error can't find the container with id 6618a76aef159a2066167066390470eb70c659262c8dff14c19d9ea0f8fe4f61 Nov 01 22:38:00.162802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.162765 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" event={"ID":"42e7221b-ee32-4b06-854e-b90394314b45","Type":"ContainerStarted","Data":"786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3"} Nov 01 22:38:00.162976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.162817 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" event={"ID":"42e7221b-ee32-4b06-854e-b90394314b45","Type":"ContainerStarted","Data":"6618a76aef159a2066167066390470eb70c659262c8dff14c19d9ea0f8fe4f61"} Nov 01 22:38:00.182105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.182059 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podStartSLOduration=1.182047153 podStartE2EDuration="1.182047153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:00.180599355 +0000 UTC m=+845.608477286" watchObservedRunningTime="2026-11-01 22:38:00.182047153 +0000 UTC m=+845.609925118" Nov 01 22:38:00.261827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.261791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:00.684884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.684851 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:38:00.685375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.685152 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:38:00.686061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.686040 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 22:38:00.743766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:00.743730 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:38:01.401625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:01.394335 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:02.123601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:02.113162 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:03.208678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:03.208651 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:04.026997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:04.026963 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:04.762309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:04.762274 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:05.551784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:05.551752 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:06.377924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:06.377892 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:07.090085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.090052 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:07.757413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.757379 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:07.849340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.849308 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924"] Nov 01 22:38:07.853801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.853782 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.872964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.872934 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924"] Nov 01 22:38:07.956877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.956846 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.956884 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkwt\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-kube-api-access-8pkwt\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.956912 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.957076 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.957144 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957255 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.957223 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.957275 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.957328 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:07.957384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:07.957363 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058239 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058297 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058338 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058365 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058390 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058487 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058525 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkwt\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-kube-api-access-8pkwt\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058565 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058647 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.058853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058827 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.059015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.058920 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.059074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.059006 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.059179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.059156 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.059283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.059265 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.060900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.060868 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.060993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.060870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.066812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.066789 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.066988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.066968 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkwt\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-kube-api-access-8pkwt\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.169047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.169016 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:08.457559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.457531 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924"] Nov 01 22:38:08.458049 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:08.458015 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6539b0b6_bb21_4d03_b7c1_c1792fcf8fea.slice/crio-5d0df423eb1c5c774bc82d34123f02f56b5217b89b7c79fea7a45c0baeee5165 WatchSource:0}: Error finding container 5d0df423eb1c5c774bc82d34123f02f56b5217b89b7c79fea7a45c0baeee5165: Status 404 returned error can't find the container with id 5d0df423eb1c5c774bc82d34123f02f56b5217b89b7c79fea7a45c0baeee5165 Nov 01 22:38:08.460719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.460683 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:08.460795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.460761 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:08.460838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.460798 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:08.543802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.543772 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7"] Nov 01 22:38:08.549465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.549438 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.589243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.589206 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7"] Nov 01 22:38:08.603970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.603933 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:08.676215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676177 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676304 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676355 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d42e2f-571f-4e4f-9913-f1510b869444-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676384 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676420 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676539 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csxch\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-kube-api-access-csxch\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676647 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d42e2f-571f-4e4f-9913-f1510b869444-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676709 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.676768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.676733 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.777925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.777845 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csxch\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-kube-api-access-csxch\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.777955 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d42e2f-571f-4e4f-9913-f1510b869444-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778004 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778029 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778226 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778372 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778484 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778522 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778691 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d42e2f-571f-4e4f-9913-f1510b869444-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778738 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778797 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d42e2f-571f-4e4f-9913-f1510b869444-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.778976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.778818 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.779117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.779091 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.779194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.779168 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.780978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.780953 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.781168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.781152 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d42e2f-571f-4e4f-9913-f1510b869444-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.787132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.787105 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.787478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.787451 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csxch\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-kube-api-access-csxch\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:08.880734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:08.880699 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:09.164520 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:09.164488 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d42e2f_571f_4e4f_9913_f1510b869444.slice/crio-ae0b92259bf917d1a2b82302dd46c34d284189a49f7c103cb5945776d213d8a8 WatchSource:0}: Error finding container ae0b92259bf917d1a2b82302dd46c34d284189a49f7c103cb5945776d213d8a8: Status 404 returned error can't find the container with id ae0b92259bf917d1a2b82302dd46c34d284189a49f7c103cb5945776d213d8a8 Nov 01 22:38:09.165699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.165674 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7"] Nov 01 22:38:09.167696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.167663 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:09.167802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.167742 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:09.167802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.167791 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:09.312191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.312162 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" event={"ID":"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea","Type":"ContainerStarted","Data":"5c0f624ec8d6a3247349a01da94c76101f2e2519d82d9231b13b674e3e59feb9"} Nov 01 22:38:09.312390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.312375 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" event={"ID":"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea","Type":"ContainerStarted","Data":"5d0df423eb1c5c774bc82d34123f02f56b5217b89b7c79fea7a45c0baeee5165"} Nov 01 22:38:09.314997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.314968 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" event={"ID":"04d42e2f-571f-4e4f-9913-f1510b869444","Type":"ContainerStarted","Data":"0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9"} Nov 01 22:38:09.315111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.315018 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" event={"ID":"04d42e2f-571f-4e4f-9913-f1510b869444","Type":"ContainerStarted","Data":"ae0b92259bf917d1a2b82302dd46c34d284189a49f7c103cb5945776d213d8a8"} Nov 01 22:38:09.335893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.335853 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podStartSLOduration=2.335841639 podStartE2EDuration="2.335841639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:09.331441752 +0000 UTC m=+854.759319685" watchObservedRunningTime="2026-11-01 22:38:09.335841639 +0000 UTC m=+854.763719609" Nov 01 22:38:09.354908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.354868 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podStartSLOduration=1.354856517 podStartE2EDuration="1.354856517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:09.351440684 +0000 UTC m=+854.779318617" watchObservedRunningTime="2026-11-01 22:38:09.354856517 +0000 UTC m=+854.782734448" Nov 01 22:38:09.474350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.474318 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:09.739273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.739182 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:38:09.751885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.751862 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:38:09.894598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:09.883666 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:10.169296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:10.169220 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:10.170311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:10.170291 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 22:38:10.170372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:10.170336 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 22:38:10.221800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:10.221755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:10.365987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:10.365950 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:38:10.903600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:10.902709 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 22:38:10.903600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:10.902775 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 22:38:11.110607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:11.110540 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:11.169652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:11.169523 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 22:38:11.169652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:11.169599 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 22:38:11.881998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:11.881962 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 22:38:11.882271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:11.882243 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 22:38:12.165418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:12.165312 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:12.169423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:12.169392 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 22:38:12.169686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:12.169652 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 22:38:12.554296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:12.554259 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:12.896658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:12.889991 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 22:38:12.896658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:12.890063 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 22:38:13.133630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.133598 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:13.170115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.170041 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 22:38:13.170115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.170099 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 22:38:13.731807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.731565 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf"] Nov 01 22:38:13.737309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.737285 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.737476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.737380 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf"] Nov 01 22:38:13.740835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.740812 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:38:13.747143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.741823 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:38:13.747143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.741912 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:38:13.747143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.742027 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:38:13.823607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.823445 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.823607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.823589 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.826014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.824302 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchhh\" (UniqueName: \"kubernetes.io/projected/13b2dd81-71a6-4bc5-88b7-bc969373fe50-kube-api-access-hchhh\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.918600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.917951 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 22:38:13.918600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.918031 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 22:38:13.926888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.926860 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.927137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.927124 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchhh\" (UniqueName: \"kubernetes.io/projected/13b2dd81-71a6-4bc5-88b7-bc969373fe50-kube-api-access-hchhh\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.927329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.927317 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.927851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.927822 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.930007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.929988 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:13.935830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:13.935812 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchhh\" (UniqueName: \"kubernetes.io/projected/13b2dd81-71a6-4bc5-88b7-bc969373fe50-kube-api-access-hchhh\") pod \"maas-api-revoke-sub-87a7b34d-bsxsf\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:14.056991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:14.056900 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:14.166091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:14.165014 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:14.170070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:14.170038 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 22:38:14.170198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:14.170093 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 22:38:14.430912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:14.429898 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf"] Nov 01 22:38:14.435016 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:14.434966 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b2dd81_71a6_4bc5_88b7_bc969373fe50.slice/crio-f9f5d863e3c0606506f191ce0350b2f94bd469cbfbf4a936670279affe92b62d WatchSource:0}: Error finding container f9f5d863e3c0606506f191ce0350b2f94bd469cbfbf4a936670279affe92b62d: Status 404 returned error can't find the container with id f9f5d863e3c0606506f191ce0350b2f94bd469cbfbf4a936670279affe92b62d Nov 01 22:38:14.550372 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:38:14.550338 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:38:14.885597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:14.883931 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 22:38:14.885597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:14.883992 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 22:38:15.068403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.068316 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:15.169611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.169550 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 01 22:38:15.169899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.169656 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 01 22:38:15.447590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.444189 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" event={"ID":"13b2dd81-71a6-4bc5-88b7-bc969373fe50","Type":"ContainerStarted","Data":"9b1d9e9f4974b483ecaa9253bccc9ad5dd08b49357ecff3c3d862d8e02596721"} Nov 01 22:38:15.447590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.444275 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" event={"ID":"13b2dd81-71a6-4bc5-88b7-bc969373fe50","Type":"ContainerStarted","Data":"f9f5d863e3c0606506f191ce0350b2f94bd469cbfbf4a936670279affe92b62d"} Nov 01 22:38:15.461125 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.461071 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" podStartSLOduration=2.461053282 podStartE2EDuration="2.461053282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:15.459354569 +0000 UTC m=+860.887232501" watchObservedRunningTime="2026-11-01 22:38:15.461053282 +0000 UTC m=+860.888931215" Nov 01 22:38:15.874229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.874140 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:15.881529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.881505 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 01 22:38:15.881665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:15.881548 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 01 22:38:16.175254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.175224 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:16.175689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.175663 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:16.176291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.176273 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:38:16.309897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.309852 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:16.466661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.465063 4981 generic.go:356] "Generic (PLEG): container finished" podID="13b2dd81-71a6-4bc5-88b7-bc969373fe50" containerID="9b1d9e9f4974b483ecaa9253bccc9ad5dd08b49357ecff3c3d862d8e02596721" exitCode=0 Nov 01 22:38:16.466661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.465635 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" event={"ID":"13b2dd81-71a6-4bc5-88b7-bc969373fe50","Type":"ContainerDied","Data":"9b1d9e9f4974b483ecaa9253bccc9ad5dd08b49357ecff3c3d862d8e02596721"} Nov 01 22:38:16.800837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.800767 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:16.886455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.886416 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:16.886932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.886911 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:16.887837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:16.887815 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:38:17.506693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.506663 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5"] Nov 01 22:38:17.515790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.515772 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.516933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.516909 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5"] Nov 01 22:38:17.591478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.591435 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.592775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.591770 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qsp\" (UniqueName: \"kubernetes.io/projected/f58f2718-f203-44c9-b9b2-514c15ed14cb-kube-api-access-v4qsp\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.592775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.591881 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.696707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.693734 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qsp\" (UniqueName: \"kubernetes.io/projected/f58f2718-f203-44c9-b9b2-514c15ed14cb-kube-api-access-v4qsp\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.696707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.693835 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.696707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.693966 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.696707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.695167 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.700976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.700946 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.705485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.705435 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qsp\" (UniqueName: \"kubernetes.io/projected/f58f2718-f203-44c9-b9b2-514c15ed14cb-kube-api-access-v4qsp\") pod \"maas-api-revoke-sub-5fc0dd06-q5zb5\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.777650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.766774 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:17.817598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.817062 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-openshift-ca\") pod \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " Nov 01 22:38:17.817598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.817300 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b2dd81-71a6-4bc5-88b7-bc969373fe50-kube-api-access-hchhh\" (UniqueName: \"kubernetes.io/projected/13b2dd81-71a6-4bc5-88b7-bc969373fe50-kube-api-access-hchhh\") pod \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " Nov 01 22:38:17.817598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.817364 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-xks-ca\") pod \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\" (UID: \"13b2dd81-71a6-4bc5-88b7-bc969373fe50\") " Nov 01 22:38:17.822598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.818183 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-openshift-ca" pod "13b2dd81-71a6-4bc5-88b7-bc969373fe50" (UID: "13b2dd81-71a6-4bc5-88b7-bc969373fe50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:17.822598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.818439 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:17.822598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.819199 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:17.850600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.847804 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:17.873609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.870460 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b2dd81-71a6-4bc5-88b7-bc969373fe50-kube-api-access-hchhh" pod "13b2dd81-71a6-4bc5-88b7-bc969373fe50" (UID: "13b2dd81-71a6-4bc5-88b7-bc969373fe50"). InnerVolumeSpecName "kube-api-access-hchhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:17.878214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.877707 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-xks-ca" pod "13b2dd81-71a6-4bc5-88b7-bc969373fe50" (UID: "13b2dd81-71a6-4bc5-88b7-bc969373fe50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:17.921173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.921132 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchhh\" (UniqueName: \"kubernetes.io/projected/13b2dd81-71a6-4bc5-88b7-bc969373fe50-kube-api-access-hchhh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:17.921173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:17.921174 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13b2dd81-71a6-4bc5-88b7-bc969373fe50-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:18.271599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:18.271411 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5"] Nov 01 22:38:18.631834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:18.631798 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" Nov 01 22:38:18.634515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:18.634484 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf" event={"ID":"13b2dd81-71a6-4bc5-88b7-bc969373fe50","Type":"ContainerDied","Data":"f9f5d863e3c0606506f191ce0350b2f94bd469cbfbf4a936670279affe92b62d"} Nov 01 22:38:18.634774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:18.634757 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9f5d863e3c0606506f191ce0350b2f94bd469cbfbf4a936670279affe92b62d" Nov 01 22:38:19.431905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:19.430008 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:38:19.446560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:19.446031 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:38:19.668478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:19.668384 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" event={"ID":"f58f2718-f203-44c9-b9b2-514c15ed14cb","Type":"ContainerStarted","Data":"0b163d04f3b42d778b6e20352e9fd72839a740546cefc2423c61c51c7362a8e6"} Nov 01 22:38:19.669069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:19.669042 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" event={"ID":"f58f2718-f203-44c9-b9b2-514c15ed14cb","Type":"ContainerStarted","Data":"48878d94f6bdf37a3578cc57bbf50fbcbe1c498d6918d8ad0f4fbc5ded106fc5"} Nov 01 22:38:20.395471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:20.395386 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:20.676310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:20.676279 4981 generic.go:356] "Generic (PLEG): container finished" podID="f58f2718-f203-44c9-b9b2-514c15ed14cb" containerID="0b163d04f3b42d778b6e20352e9fd72839a740546cefc2423c61c51c7362a8e6" exitCode=0 Nov 01 22:38:20.676732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:20.676341 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" event={"ID":"f58f2718-f203-44c9-b9b2-514c15ed14cb","Type":"ContainerDied","Data":"0b163d04f3b42d778b6e20352e9fd72839a740546cefc2423c61c51c7362a8e6"} Nov 01 22:38:21.254273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.254239 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:21.831261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.831236 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:21.965417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.965333 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:21.969449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.969424 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-openshift-ca\") pod \"f58f2718-f203-44c9-b9b2-514c15ed14cb\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " Nov 01 22:38:21.969611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.969485 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58f2718-f203-44c9-b9b2-514c15ed14cb-kube-api-access-v4qsp\" (UniqueName: \"kubernetes.io/projected/f58f2718-f203-44c9-b9b2-514c15ed14cb-kube-api-access-v4qsp\") pod \"f58f2718-f203-44c9-b9b2-514c15ed14cb\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " Nov 01 22:38:21.969755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.969737 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-xks-ca\") pod \"f58f2718-f203-44c9-b9b2-514c15ed14cb\" (UID: \"f58f2718-f203-44c9-b9b2-514c15ed14cb\") " Nov 01 22:38:21.969809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.969772 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-openshift-ca" pod "f58f2718-f203-44c9-b9b2-514c15ed14cb" (UID: "f58f2718-f203-44c9-b9b2-514c15ed14cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:21.970341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.970324 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:21.971559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.971538 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58f2718-f203-44c9-b9b2-514c15ed14cb-kube-api-access-v4qsp" pod "f58f2718-f203-44c9-b9b2-514c15ed14cb" (UID: "f58f2718-f203-44c9-b9b2-514c15ed14cb"). InnerVolumeSpecName "kube-api-access-v4qsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:21.971683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:21.971658 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-xks-ca" pod "f58f2718-f203-44c9-b9b2-514c15ed14cb" (UID: "f58f2718-f203-44c9-b9b2-514c15ed14cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:22.071793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.071761 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f58f2718-f203-44c9-b9b2-514c15ed14cb-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:22.071793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.071791 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qsp\" (UniqueName: \"kubernetes.io/projected/f58f2718-f203-44c9-b9b2-514c15ed14cb-kube-api-access-v4qsp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:22.693043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.693012 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" Nov 01 22:38:22.693212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.693008 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5" event={"ID":"f58f2718-f203-44c9-b9b2-514c15ed14cb","Type":"ContainerDied","Data":"48878d94f6bdf37a3578cc57bbf50fbcbe1c498d6918d8ad0f4fbc5ded106fc5"} Nov 01 22:38:22.693212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.693173 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48878d94f6bdf37a3578cc57bbf50fbcbe1c498d6918d8ad0f4fbc5ded106fc5" Nov 01 22:38:22.763292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.763262 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:22.997782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.997710 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f"] Nov 01 22:38:22.998330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.998316 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f58f2718-f203-44c9-b9b2-514c15ed14cb" containerName="revoke-keys" Nov 01 22:38:22.998374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.998332 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58f2718-f203-44c9-b9b2-514c15ed14cb" containerName="revoke-keys" Nov 01 22:38:22.998374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.998350 4981 cpu_manager.go:414] "containerMap: removing container" podUID="13b2dd81-71a6-4bc5-88b7-bc969373fe50" containerName="revoke-keys" Nov 01 22:38:22.998374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.998355 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b2dd81-71a6-4bc5-88b7-bc969373fe50" containerName="revoke-keys" Nov 01 22:38:22.998468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.998447 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58f2718-f203-44c9-b9b2-514c15ed14cb" containerName="revoke-keys" Nov 01 22:38:22.998468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:22.998458 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b2dd81-71a6-4bc5-88b7-bc969373fe50" containerName="revoke-keys" Nov 01 22:38:23.002241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.002205 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.019682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.019651 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f"] Nov 01 22:38:23.085755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.085714 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.085939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.085777 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.086008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.085982 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.086060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.086022 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.086060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.086048 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.086120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.086091 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.086196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.086175 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.086280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.086264 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbd2j\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-kube-api-access-hbd2j\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.086335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.086320 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187428 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbd2j\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-kube-api-access-hbd2j\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187472 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187525 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187549 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187737 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187773 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187801 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187846 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.187979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.187876 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.188049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.188003 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.188387 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.188360 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.188505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.188363 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.188505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.188414 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.188675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.188653 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.190299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.190276 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.190434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.190410 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.198413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.198389 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbd2j\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-kube-api-access-hbd2j\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.198872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.198849 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.318717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.318637 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:23.449488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.449452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:23.615757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.615729 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f"] Nov 01 22:38:23.616104 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:23.616078 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c07ded_c806_40fb_88af_7ea7c2648bdd.slice/crio-3986f8c392322ba4fe95be8af6482d84529578bdd041ca2528966e6fe123c777 WatchSource:0}: Error finding container 3986f8c392322ba4fe95be8af6482d84529578bdd041ca2528966e6fe123c777: Status 404 returned error can't find the container with id 3986f8c392322ba4fe95be8af6482d84529578bdd041ca2528966e6fe123c777 Nov 01 22:38:23.619117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.619082 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:23.619190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.619165 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:23.619234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.619209 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:23.702141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.702001 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" event={"ID":"e8c07ded-c806-40fb-88af-7ea7c2648bdd","Type":"ContainerStarted","Data":"f1a201036d970e1107ee0def481a18fd4a5f13a5d4c23552ce17a84bdb3af3a2"} Nov 01 22:38:23.702141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.702066 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" event={"ID":"e8c07ded-c806-40fb-88af-7ea7c2648bdd","Type":"ContainerStarted","Data":"3986f8c392322ba4fe95be8af6482d84529578bdd041ca2528966e6fe123c777"} Nov 01 22:38:23.729877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.729831 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podStartSLOduration=1.72981702 podStartE2EDuration="1.72981702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:23.727290786 +0000 UTC m=+869.155168718" watchObservedRunningTime="2026-11-01 22:38:23.72981702 +0000 UTC m=+869.157694967" Nov 01 22:38:23.780119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.780087 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6"] Nov 01 22:38:23.785283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.785254 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.796473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.796445 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6"] Nov 01 22:38:23.897673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.897588 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e27162-6859-45a7-a557-8e9ddd63c463-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.897828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.897678 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.897828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.897788 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.897956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.897902 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.898048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.898029 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6lf\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-kube-api-access-sk6lf\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.898104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.898070 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.898200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.898186 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e27162-6859-45a7-a557-8e9ddd63c463-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.898285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.898244 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.898343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.898283 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:23.958848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.958815 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb"] Nov 01 22:38:23.963761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.963732 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:23.978743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.978709 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb"] Nov 01 22:38:23.999976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:23.999949 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000015 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6lf\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-kube-api-access-sk6lf\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000043 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000078 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e27162-6859-45a7-a557-8e9ddd63c463-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000105 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000143 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000273 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e27162-6859-45a7-a557-8e9ddd63c463-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000323 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000373 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000437 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000633 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.000868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000804 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.001026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.000879 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.001402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.001376 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e27162-6859-45a7-a557-8e9ddd63c463-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.002933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.002909 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.003816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.003789 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e27162-6859-45a7-a557-8e9ddd63c463-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.008670 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.008643 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.008756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.008704 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6lf\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-kube-api-access-sk6lf\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.101234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101199 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101255 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101338 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101403 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101567 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101696 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101717 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zv8\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-kube-api-access-b7zv8\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.101761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.101736 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.109915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.109892 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:24.214494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.214457 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.214706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.214545 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.214706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.214595 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.214844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.214795 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.214981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.214960 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215014 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215104 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215147 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zv8\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-kube-api-access-b7zv8\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215186 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215496 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215674 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215825 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.215995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.215974 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.216273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.216249 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.223884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.217528 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.224805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.224778 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.227181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.227155 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.229072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.229048 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zv8\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-kube-api-access-b7zv8\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.285406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.285382 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:24.319892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.319862 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:24.321307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.321285 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:38:24.321406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.321329 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:38:24.323524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.323486 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:24.396522 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:24.396470 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e27162_6859_45a7_a557_8e9ddd63c463.slice/crio-930b1c585b32c00cab0ca72cd61b53cd55b61ddd263e2520a032ceb83a96ae0e WatchSource:0}: Error finding container 930b1c585b32c00cab0ca72cd61b53cd55b61ddd263e2520a032ceb83a96ae0e: Status 404 returned error can't find the container with id 930b1c585b32c00cab0ca72cd61b53cd55b61ddd263e2520a032ceb83a96ae0e Nov 01 22:38:24.400144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.400117 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6"] Nov 01 22:38:24.404941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.404887 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:24.405065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.404980 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:24.405065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.405023 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:24.571534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.571497 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb"] Nov 01 22:38:24.587068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.587027 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:24.587282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.587263 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:24.587441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.587424 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:24.716453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.716366 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" event={"ID":"2a82c996-36f9-4a94-803b-f78f1eaa8bdd","Type":"ContainerStarted","Data":"629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6"} Nov 01 22:38:24.716453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.716421 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" event={"ID":"2a82c996-36f9-4a94-803b-f78f1eaa8bdd","Type":"ContainerStarted","Data":"afaa2cdff6920c15efef0976bcd2d5ac29a2a56542949ceabf9e170457e44ea6"} Nov 01 22:38:24.718289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.718259 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" event={"ID":"64e27162-6859-45a7-a557-8e9ddd63c463","Type":"ContainerStarted","Data":"e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5"} Nov 01 22:38:24.718427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.718316 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" event={"ID":"64e27162-6859-45a7-a557-8e9ddd63c463","Type":"ContainerStarted","Data":"930b1c585b32c00cab0ca72cd61b53cd55b61ddd263e2520a032ceb83a96ae0e"} Nov 01 22:38:24.739822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.737465 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podStartSLOduration=1.7374507609999998 podStartE2EDuration="1.737450761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:24.735511499 +0000 UTC m=+870.163389434" watchObservedRunningTime="2026-11-01 22:38:24.737450761 +0000 UTC m=+870.165328692" Nov 01 22:38:24.766163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:24.766112 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podStartSLOduration=1.766097311 podStartE2EDuration="1.766097311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:24.760960349 +0000 UTC m=+870.188838281" watchObservedRunningTime="2026-11-01 22:38:24.766097311 +0000 UTC m=+870.193975323" Nov 01 22:38:25.110187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.110093 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:25.111885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.111857 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:38:25.112022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.111907 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 22:38:25.263224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.263156 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:25.296606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.292742 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:25.305652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.302501 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:38:25.305652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.302601 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 22:38:25.327601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.320309 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:38:25.327601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:25.320383 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:38:26.136602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.134621 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd"] Nov 01 22:38:26.162601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.157337 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.162601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.157397 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 22:38:26.193600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.178639 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:26.255641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.248766 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd"] Nov 01 22:38:26.255641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.248892 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.268236 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.268177 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-home\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.268417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.268256 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-model-cache\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.268417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.268337 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d65681-0a07-4326-be1a-8df4c0fe831d-tls-certs\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.268417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.268368 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-tmp-dir\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.268630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.268430 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-dshm\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.268630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.268545 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdp66\" (UniqueName: \"kubernetes.io/projected/a9d65681-0a07-4326-be1a-8df4c0fe831d-kube-api-access-sdp66\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.329988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.323922 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2f3705\"/\"distinct-2-w4-2f3705-kserve-self-signed-certs\"" Nov 01 22:38:26.329988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.325404 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.329988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.325464 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:38:26.329988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.326070 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:38:26.329988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.326111 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.388825 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-model-cache\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.388942 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d65681-0a07-4326-be1a-8df4c0fe831d-tls-certs\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.388981 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-tmp-dir\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.389061 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-dshm\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.389198 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdp66\" (UniqueName: \"kubernetes.io/projected/a9d65681-0a07-4326-be1a-8df4c0fe831d-kube-api-access-sdp66\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.389253 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-home\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.389868 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-home\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.398660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.390469 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-model-cache\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.421375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.421333 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-tmp-dir\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.472602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.469698 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d65681-0a07-4326-be1a-8df4c0fe831d-tls-certs\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.510813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.510774 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-dshm\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.605599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.588296 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdp66\" (UniqueName: \"kubernetes.io/projected/a9d65681-0a07-4326-be1a-8df4c0fe831d-kube-api-access-sdp66\") pod \"distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:26.809671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:26.793353 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:27.114674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.114011 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.114674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.114075 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 22:38:27.156461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.156427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:27.299596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.296814 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.299596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.296886 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 22:38:27.358615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.357403 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:38:27.358615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.357462 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:38:27.464727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.464692 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd"] Nov 01 22:38:27.471187 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:27.471164 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9d65681_0a07_4326_be1a_8df4c0fe831d.slice/crio-6a817ce4332db83f4d4275596876b853fe59e4d8f56ae850a60913f11565b7fc WatchSource:0}: Error finding container 6a817ce4332db83f4d4275596876b853fe59e4d8f56ae850a60913f11565b7fc: Status 404 returned error can't find the container with id 6a817ce4332db83f4d4275596876b853fe59e4d8f56ae850a60913f11565b7fc Nov 01 22:38:27.853151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:27.853098 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" event={"ID":"a9d65681-0a07-4326-be1a-8df4c0fe831d","Type":"ContainerStarted","Data":"6a817ce4332db83f4d4275596876b853fe59e4d8f56ae850a60913f11565b7fc"} Nov 01 22:38:28.114213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.114115 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.114497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.114451 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 22:38:28.296601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.285961 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.296601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.286024 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 22:38:28.328601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.319096 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:38:28.328601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.319153 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:38:28.553985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.553947 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:28.862527 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.862434 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" event={"ID":"a9d65681-0a07-4326-be1a-8df4c0fe831d","Type":"ContainerStarted","Data":"4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5"} Nov 01 22:38:28.900600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:28.899729 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podStartSLOduration=2.899708936 podStartE2EDuration="2.899708936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:28.887953676 +0000 UTC m=+874.315831610" watchObservedRunningTime="2026-11-01 22:38:28.899708936 +0000 UTC m=+874.327586868" Nov 01 22:38:29.130658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.119699 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.130658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.119765 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 22:38:29.289603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.285950 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.289603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.286011 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 22:38:29.328602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.328040 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:38:29.328602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.328101 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:38:29.662610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.662017 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:29.754675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.754447 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:29.796705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:29.796675 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:30.114052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:30.114014 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.114354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:30.114320 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 22:38:30.287600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:30.286950 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.287600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:30.287010 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 22:38:30.330794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:30.330667 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 01 22:38:30.331428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:30.331399 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 01 22:38:31.059731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.059695 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:31.111273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.111230 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 01 22:38:31.111469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.111296 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 01 22:38:31.285751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.285719 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 01 22:38:31.285897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.285773 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 01 22:38:31.524360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.524325 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:31.525075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.525053 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:31.525996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.525973 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:38:31.970558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:31.970524 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:32.116711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:32.116680 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:32.117832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:32.117805 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:32.118620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:32.118565 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:38:32.575343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:32.575309 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:32.576134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:32.576115 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:32.577023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:32.577000 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:38:32.876488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:32.876398 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:33.154901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.154817 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4"] Nov 01 22:38:33.159460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.159439 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.170679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.170657 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4"] Nov 01 22:38:33.171119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.171104 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e26733\"/\"distinct-2-w5-e26733-kserve-self-signed-certs\"" Nov 01 22:38:33.222717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.222685 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tls-certs\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.222870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.222724 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-model-cache\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.222918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.222896 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tmp-dir\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.222956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.222937 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jqn\" (UniqueName: \"kubernetes.io/projected/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-kube-api-access-n4jqn\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.223022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.223003 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-home\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.223067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.223056 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-dshm\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.230004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.229981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:33.325665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.325397 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-home\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.325665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.325472 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-dshm\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.325665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.325666 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tls-certs\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.326141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.325703 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-model-cache\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.326141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.325834 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tmp-dir\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.326141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.325879 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jqn\" (UniqueName: \"kubernetes.io/projected/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-kube-api-access-n4jqn\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.326141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.325908 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-home\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.326535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.326511 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-model-cache\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.326934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.326913 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tmp-dir\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.345369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.345311 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tls-certs\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.348435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.348396 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jqn\" (UniqueName: \"kubernetes.io/projected/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-kube-api-access-n4jqn\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.356043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.355983 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-dshm\") pod \"distinct-2-w5-e26733-kserve-59944cd7d5-g49x4\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.474609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.474544 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:33.759988 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:33.759956 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcea13bb9_9fdf_448e_b15f_a42b48ea6edb.slice/crio-27e49209ad7f1df041e5ccdd835f379a0cdc3971be27c06474ebbf6ee84473f7 WatchSource:0}: Error finding container 27e49209ad7f1df041e5ccdd835f379a0cdc3971be27c06474ebbf6ee84473f7: Status 404 returned error can't find the container with id 27e49209ad7f1df041e5ccdd835f379a0cdc3971be27c06474ebbf6ee84473f7 Nov 01 22:38:33.760782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.760753 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4"] Nov 01 22:38:33.790473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:33.790450 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:34.052131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:34.052038 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" event={"ID":"cea13bb9-9fdf-448e-b15f-a42b48ea6edb","Type":"ContainerStarted","Data":"2bb0195e003c7f78c178520adb0382ee893bd022ae3d9ff01f0f53bd1517e665"} Nov 01 22:38:34.052131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:34.052082 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" event={"ID":"cea13bb9-9fdf-448e-b15f-a42b48ea6edb","Type":"ContainerStarted","Data":"27e49209ad7f1df041e5ccdd835f379a0cdc3971be27c06474ebbf6ee84473f7"} Nov 01 22:38:34.071545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:34.071499 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podStartSLOduration=1.071483989 podStartE2EDuration="1.071483989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:34.068775737 +0000 UTC m=+879.496653666" watchObservedRunningTime="2026-11-01 22:38:34.071483989 +0000 UTC m=+879.499361954" Nov 01 22:38:34.475219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:34.475190 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:34.559439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:34.559410 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:35.470498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:35.470461 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:36.208318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:36.208281 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:36.793970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:36.793941 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:36.806696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:36.806672 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:36.880701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:36.880674 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:37.087648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.087547 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:37.647700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.647667 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:37.875223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.875193 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml"] Nov 01 22:38:37.879799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.879782 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:37.884023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.884002 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:38:37.885048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.885029 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:38:37.885180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.885160 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:38:37.886765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.886720 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:38:37.887124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.887099 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml"] Nov 01 22:38:37.959466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.959434 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz"] Nov 01 22:38:37.965070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.965042 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:37.984133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.984108 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz"] Nov 01 22:38:37.990753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.990727 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:37.990877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.990791 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9x6\" (UniqueName: \"kubernetes.io/projected/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-kube-api-access-kj9x6\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:37.990940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:37.990891 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.092230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092194 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkvr\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-kube-api-access-wfkvr\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092245 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092309 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092361 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092415 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.092533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092435 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092513 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092552 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092668 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9x6\" (UniqueName: \"kubernetes.io/projected/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-kube-api-access-kj9x6\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.092737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092706 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.092737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092727 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.092874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.092855 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.093106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.093088 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.095022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.095003 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.106037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.106013 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9x6\" (UniqueName: \"kubernetes.io/projected/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-kube-api-access-kj9x6\") pod \"maas-api-revoke-sub-289483c7-khlml\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.193883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.193848 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkvr\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-kube-api-access-wfkvr\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.193915 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.193952 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194012 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194058 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194087 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194131 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194371 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194484 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194625 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194690 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-data\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194845 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-certs\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.194986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.194939 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istiod-ca-cert\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.195110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.195084 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-credential-socket\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.196532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.196509 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-envoy\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.196669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.196632 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-podinfo\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.196755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.196714 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:38.204226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.204196 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-token\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.204637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.204620 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkvr\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-kube-api-access-wfkvr\") pod \"e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.284848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.284795 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:38.482134 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:38.482101 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f00b8dd_9abd_4b90_9383_bbd52bb4bc88.slice/crio-3892d760b4faa321364304a66b902bafbfb85067fb04d714e726c0be01817a13 WatchSource:0}: Error finding container 3892d760b4faa321364304a66b902bafbfb85067fb04d714e726c0be01817a13: Status 404 returned error can't find the container with id 3892d760b4faa321364304a66b902bafbfb85067fb04d714e726c0be01817a13 Nov 01 22:38:38.482134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.482114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:38.482701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.482682 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml"] Nov 01 22:38:38.585837 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:38.585806 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8d8482_22c2_4804_a7db_fe1dbcf168d7.slice/crio-edb71e9c532a8016c3253559d0775ad02744bba9d751de1ebe57bc5c019230c6 WatchSource:0}: Error finding container edb71e9c532a8016c3253559d0775ad02744bba9d751de1ebe57bc5c019230c6: Status 404 returned error can't find the container with id edb71e9c532a8016c3253559d0775ad02744bba9d751de1ebe57bc5c019230c6 Nov 01 22:38:38.588167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.588142 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz"] Nov 01 22:38:38.588624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.588561 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:38.588687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.588656 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:38.588687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.588684 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:38.851561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.851477 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs"] Nov 01 22:38:38.855948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.855923 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:38.873029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:38.873005 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs"] Nov 01 22:38:39.006608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.006553 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.006806 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.006883 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.006975 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.007047 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.007090 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.007292 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjh5t\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-kube-api-access-qjh5t\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.007365 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.007456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.007441 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.101256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.101214 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" event={"ID":"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88","Type":"ContainerStarted","Data":"e4a54a3aa5587ac5f45f123fd0942f176d06595639b96dcc3588789f579cb5b1"} Nov 01 22:38:39.101423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.101310 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" event={"ID":"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88","Type":"ContainerStarted","Data":"3892d760b4faa321364304a66b902bafbfb85067fb04d714e726c0be01817a13"} Nov 01 22:38:39.103318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.103248 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" event={"ID":"ee8d8482-22c2-4804-a7db-fe1dbcf168d7","Type":"ContainerStarted","Data":"1afc26661d7f405df2f5cfacdf5416d5432e14c66f6ba350aef96d3f412eb864"} Nov 01 22:38:39.103318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.103288 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" event={"ID":"ee8d8482-22c2-4804-a7db-fe1dbcf168d7","Type":"ContainerStarted","Data":"edb71e9c532a8016c3253559d0775ad02744bba9d751de1ebe57bc5c019230c6"} Nov 01 22:38:39.108779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.108753 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.108887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.108796 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.108887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.108822 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.108887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.108860 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.108887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.108883 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.108969 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjh5t\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-kube-api-access-qjh5t\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109012 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109048 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109079 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109273 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-data\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109349 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109656 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-credential-socket\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109657 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istiod-ca-cert\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.109791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.109760 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-certs\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.111192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.111168 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-envoy\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.111365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.111343 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-podinfo\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.117750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.117727 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-token\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.118875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.118830 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjh5t\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-kube-api-access-qjh5t\") pod \"e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.122359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.122312 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" podStartSLOduration=2.122297071 podStartE2EDuration="2.122297071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:39.119545534 +0000 UTC m=+884.547423467" watchObservedRunningTime="2026-11-01 22:38:39.122297071 +0000 UTC m=+884.550175004" Nov 01 22:38:39.159632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.155460 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podStartSLOduration=2.155444269 podStartE2EDuration="2.155444269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:39.149242843 +0000 UTC m=+884.577120774" watchObservedRunningTime="2026-11-01 22:38:39.155444269 +0000 UTC m=+884.583322202" Nov 01 22:38:39.172621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.172393 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:39.285535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.285505 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:39.286944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.286922 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:38:39.287056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.286979 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 22:38:39.348933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.348896 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:39.471118 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:39.471088 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3538e3c0_6abd_42ce_9508_06fbf69b6e48.slice/crio-1770e77d2fa6cff9516cac3cbde47cd84acc3ace984537a6a04149e3e952e3e6 WatchSource:0}: Error finding container 1770e77d2fa6cff9516cac3cbde47cd84acc3ace984537a6a04149e3e952e3e6: Status 404 returned error can't find the container with id 1770e77d2fa6cff9516cac3cbde47cd84acc3ace984537a6a04149e3e952e3e6 Nov 01 22:38:39.472360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.472336 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs"] Nov 01 22:38:39.493598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.474292 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:39.493598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.474378 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:39.493598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:39.474422 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:40.113878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.113541 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" event={"ID":"3538e3c0-6abd-42ce-9508-06fbf69b6e48","Type":"ContainerStarted","Data":"54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1"} Nov 01 22:38:40.113878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.113635 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" event={"ID":"3538e3c0-6abd-42ce-9508-06fbf69b6e48","Type":"ContainerStarted","Data":"1770e77d2fa6cff9516cac3cbde47cd84acc3ace984537a6a04149e3e952e3e6"} Nov 01 22:38:40.115987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.115964 4981 generic.go:356] "Generic (PLEG): container finished" podID="7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" containerID="e4a54a3aa5587ac5f45f123fd0942f176d06595639b96dcc3588789f579cb5b1" exitCode=0 Nov 01 22:38:40.116687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.116661 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" event={"ID":"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88","Type":"ContainerDied","Data":"e4a54a3aa5587ac5f45f123fd0942f176d06595639b96dcc3588789f579cb5b1"} Nov 01 22:38:40.139342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.139293 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podStartSLOduration=2.13928033 podStartE2EDuration="2.13928033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:40.136011692 +0000 UTC m=+885.563889636" watchObservedRunningTime="2026-11-01 22:38:40.13928033 +0000 UTC m=+885.567158260" Nov 01 22:38:40.173424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.173393 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:40.193598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.189022 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:38:40.193598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.189086 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 22:38:40.241688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.241653 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:40.286600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.286022 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:38:40.286600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:40.286086 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 22:38:41.173486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.173452 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.174008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.173510 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 22:38:41.242944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.242148 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:41.297602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.286079 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:38:41.297602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.286151 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 22:38:41.435900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.435869 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:41.459604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.458382 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-xks-ca\") pod \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " Nov 01 22:38:41.459604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.458501 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-openshift-ca\") pod \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " Nov 01 22:38:41.459604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.458614 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-kube-api-access-kj9x6\" (UniqueName: \"kubernetes.io/projected/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-kube-api-access-kj9x6\") pod \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\" (UID: \"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88\") " Nov 01 22:38:41.459604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.458966 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-openshift-ca" pod "7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" (UID: "7f00b8dd-9abd-4b90-9383-bbd52bb4bc88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:41.459604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.459374 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:41.462667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.461271 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-xks-ca" pod "7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" (UID: "7f00b8dd-9abd-4b90-9383-bbd52bb4bc88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:41.462667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.461596 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-kube-api-access-kj9x6" pod "7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" (UID: "7f00b8dd-9abd-4b90-9383-bbd52bb4bc88"). InnerVolumeSpecName "kube-api-access-kj9x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:41.560746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.560711 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:41.560951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:41.560931 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj9x6\" (UniqueName: \"kubernetes.io/projected/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88-kube-api-access-kj9x6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:42.146615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.146570 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" Nov 01 22:38:42.146796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.146691 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml" event={"ID":"7f00b8dd-9abd-4b90-9383-bbd52bb4bc88","Type":"ContainerDied","Data":"3892d760b4faa321364304a66b902bafbfb85067fb04d714e726c0be01817a13"} Nov 01 22:38:42.146796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.146727 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3892d760b4faa321364304a66b902bafbfb85067fb04d714e726c0be01817a13" Nov 01 22:38:42.173168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.173145 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.173291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.173184 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 22:38:42.300600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.299823 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:38:42.300600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.299881 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 22:38:42.603536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:42.603436 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:43.173159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.173128 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.173354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.173179 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 22:38:43.295032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.294991 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:38:43.295268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.295239 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 22:38:43.485011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.484934 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:43.550146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.550105 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn"] Nov 01 22:38:43.551528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.551501 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" containerName="revoke-keys" Nov 01 22:38:43.551725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.551710 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" containerName="revoke-keys" Nov 01 22:38:43.552087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.552071 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" containerName="revoke-keys" Nov 01 22:38:43.555608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.555491 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:43.557317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.557299 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn"] Nov 01 22:38:43.557586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.557506 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.558721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.558699 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:43.562180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.562151 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6ac182\"/\"distinct-2-w3-6ac182-kserve-self-signed-certs\"" Nov 01 22:38:43.645343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.645305 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tmp-dir\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.645508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.645353 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtj8l\" (UniqueName: \"kubernetes.io/projected/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-kube-api-access-dtj8l\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.645508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.645425 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-model-cache\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.645508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.645478 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-home\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.645508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.645503 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tls-certs\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.645734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.645655 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-dshm\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.760202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.760107 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-home\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.760459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.760434 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tls-certs\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.760801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.760781 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-dshm\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.760966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.760951 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tmp-dir\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.761083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.761067 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtj8l\" (UniqueName: \"kubernetes.io/projected/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-kube-api-access-dtj8l\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.761307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.761292 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-model-cache\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.762272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.762250 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-home\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.764449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.762963 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tmp-dir\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.764449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.764410 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-model-cache\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.767951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.767891 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tls-certs\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.768217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.768196 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-dshm\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.794602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.778181 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtj8l\" (UniqueName: \"kubernetes.io/projected/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-kube-api-access-dtj8l\") pod \"distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:43.935464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:43.935423 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:44.207599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:44.197951 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.207599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:44.198010 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 22:38:44.229881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:44.229846 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:44.269283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:44.269254 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn"] Nov 01 22:38:44.314201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:44.314169 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:38:44.338398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:44.338351 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 22:38:44.428321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:44.425905 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:44.612073 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:38:44.611916 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:38:45.013697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.013662 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2"] Nov 01 22:38:45.018119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.018096 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.025635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.025613 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:38:45.026651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.026413 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:38:45.027043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.027023 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:38:45.027441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.027423 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:38:45.041984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.041955 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2"] Nov 01 22:38:45.153345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.153310 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9nc\" (UniqueName: \"kubernetes.io/projected/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-kube-api-access-mp9nc\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.153514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.153408 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.153514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.153450 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.175605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.173274 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.175605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.173354 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 22:38:45.184599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.182654 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" event={"ID":"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0","Type":"ContainerStarted","Data":"642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66"} Nov 01 22:38:45.184599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.182698 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" event={"ID":"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0","Type":"ContainerStarted","Data":"392c4a1368f59e9496ee80645439540c050a6dd7ac55002cf90d368b7a05d52d"} Nov 01 22:38:45.243264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.243216 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podStartSLOduration=2.243203341 podStartE2EDuration="2.243203341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:45.239819589 +0000 UTC m=+890.667697513" watchObservedRunningTime="2026-11-01 22:38:45.243203341 +0000 UTC m=+890.671081319" Nov 01 22:38:45.257356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.257325 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9nc\" (UniqueName: \"kubernetes.io/projected/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-kube-api-access-mp9nc\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.257512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.257390 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.257512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.257423 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.258537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.258504 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.263080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.263056 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.303643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.288917 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 22:38:45.303643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.288993 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 22:38:45.310515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.310485 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9nc\" (UniqueName: \"kubernetes.io/projected/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-kube-api-access-mp9nc\") pod \"maas-api-revoke-sub-4150dc57-twhd2\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.333901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.333659 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:45.937600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:45.936453 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:46.015603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:46.010862 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2"] Nov 01 22:38:46.188923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:46.174406 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 01 22:38:46.188923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:46.174475 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 01 22:38:47.114557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.114411 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.116156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.116123 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.147545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.147311 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.200600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.195996 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:38:47.258706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.258658 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.334217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.333730 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:47.334217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.333800 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.429600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.427866 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.492704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.492656 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:38:47.619810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.499131 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.658907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.500257 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:38:47.658907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.500304 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.658907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.500330 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:38:47.658907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.629916 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.658907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.633645 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.679632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.675024 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:47.679632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.675100 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.812601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.811809 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:47.918374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.918282 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" event={"ID":"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8","Type":"ContainerStarted","Data":"1d93497e3db6068ba20b6440bdaf5d4e477522f9b6600f614ef46d0b884ae246"} Nov 01 22:38:47.918658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.918631 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" event={"ID":"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8","Type":"ContainerStarted","Data":"d791b66a29859415b7ef6be3584d2d4b14f34db7e50568016532a2781a49924f"} Nov 01 22:38:47.978733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.978683 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:48.011633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:47.986666 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.012047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.011992 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.117609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.116657 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:38:48.121608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.118764 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:38:48.121608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.120011 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.134609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.133884 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:48.287738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.220672 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:38:48.287738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.252821 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:38:48.287738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.253179 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:48.287738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.253255 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:38:48.392000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.366564 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:48.392000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.366644 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.530602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.499833 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:38:48.530602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.501161 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:38:48.547652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.533678 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:48.547652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.533735 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.548013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.547969 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:38:48.562494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.550244 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:38:48.649600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.617817 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:38:48.649600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.618410 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:38:48.686011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.669059 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:38:48.725646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.686455 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:38:48.725646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.717826 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.725646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.718392 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:38:48.725646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.719287 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:38:48.725646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.719345 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:38:48.725646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.720861 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.742731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.727061 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:48.743659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.743623 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:38:48.793073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:48.748834 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:38:49.122694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.122634 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.135674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.132530 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:38:49.135674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.133440 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:38:49.321600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.313395 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.321600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.313785 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:38:49.547600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.542000 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.547600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.542108 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:49.547600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.542136 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.562597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.561644 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:38:49.637606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.636762 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.641398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.641366 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:38:49.641718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.641699 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:49.649756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.649731 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:38:49.649984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.649965 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:38:49.650143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.650129 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:38:49.650285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.650271 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:38:49.711290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.711246 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.729717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.729665 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.731031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.730636 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:49.731031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.730715 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.731318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.731294 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:49.749892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.749849 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:38:49.755651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.755623 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:38:49.759776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:49.759656 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:50.122697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.122652 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:38:50.158937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.158885 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:38:50.175737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.175705 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:38:50.214897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.214865 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" podUID="7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:50.361136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.361096 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:38:50.385735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.385505 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:38:50.388872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.388850 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:38:50.397130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.397107 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:38:50.397347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.397331 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:38:50.397514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.397501 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:38:50.406963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.406938 4981 generic.go:356] "Generic (PLEG): container finished" podID="c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" containerID="1d93497e3db6068ba20b6440bdaf5d4e477522f9b6600f614ef46d0b884ae246" exitCode=0 Nov 01 22:38:50.407564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.407417 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:50.422782 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:38:50.422754 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20d2e0c_8a55_42f5_adf8_71d8bb3a8fc8.slice/crio-1d93497e3db6068ba20b6440bdaf5d4e477522f9b6600f614ef46d0b884ae246.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:38:50.465602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.422961 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:38:50.465602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.444052 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" event={"ID":"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8","Type":"ContainerDied","Data":"1d93497e3db6068ba20b6440bdaf5d4e477522f9b6600f614ef46d0b884ae246"} Nov 01 22:38:50.465602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.444127 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:38:50.472554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.472531 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:38:50.553992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.553964 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:38:50.618835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.617339 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:50.618835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.618673 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:50.656983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.656869 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:38:50.714089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.714015 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:38:50.714089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:50.714080 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:38:51.416003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.415980 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:38:51.607672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.607647 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:51.670233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.670157 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-kube-api-access-mp9nc\" (UniqueName: \"kubernetes.io/projected/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-kube-api-access-mp9nc\") pod \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " Nov 01 22:38:51.670233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.670209 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-openshift-ca\") pod \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " Nov 01 22:38:51.670429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.670405 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-xks-ca\") pod \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\" (UID: \"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8\") " Nov 01 22:38:51.670685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.670662 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-openshift-ca" pod "c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" (UID: "c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:51.671087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.671066 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:51.672399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.672376 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-xks-ca" pod "c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" (UID: "c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:51.672476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.672398 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-kube-api-access-mp9nc" pod "c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" (UID: "c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8"). InnerVolumeSpecName "kube-api-access-mp9nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:51.772801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.772772 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9nc\" (UniqueName: \"kubernetes.io/projected/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-kube-api-access-mp9nc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:51.772801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:51.772798 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:52.257855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:52.257822 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:52.423720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:52.423690 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" Nov 01 22:38:52.424092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:52.423727 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2" event={"ID":"c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8","Type":"ContainerDied","Data":"d791b66a29859415b7ef6be3584d2d4b14f34db7e50568016532a2781a49924f"} Nov 01 22:38:52.424092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:52.423777 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d791b66a29859415b7ef6be3584d2d4b14f34db7e50568016532a2781a49924f" Nov 01 22:38:53.110378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.110337 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts"] Nov 01 22:38:53.111077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.111055 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" containerName="revoke-keys" Nov 01 22:38:53.111193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.111077 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" containerName="revoke-keys" Nov 01 22:38:53.111319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.111305 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" containerName="revoke-keys" Nov 01 22:38:53.115137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.115114 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.130238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.130212 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts"] Nov 01 22:38:53.193345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193314 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193356 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193453 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k96hc\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-kube-api-access-k96hc\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193568 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193731 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193805 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193844 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193868 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.193960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.193930 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.285956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.285913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:53.295413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295375 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.295621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295471 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.295621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295510 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.295621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295530 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.295621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295556 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.295857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295627 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.295857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295759 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.295857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295805 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.296006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.295878 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k96hc\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-kube-api-access-k96hc\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.296347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.296315 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.296559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.296528 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.296766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.296739 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.296837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.296769 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.297032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.297002 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.298080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.298051 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.298328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.298309 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.307782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.307753 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k96hc\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-kube-api-access-k96hc\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.307913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.307892 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.433478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.433446 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:53.514768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.514735 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh"] Nov 01 22:38:53.520425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.520406 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.528132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.527919 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:38:53.528132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.527998 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:38:53.528132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.528056 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:38:53.528619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.528532 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:38:53.532452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.531925 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh"] Nov 01 22:38:53.601265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.601228 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bbq\" (UniqueName: \"kubernetes.io/projected/ea0940e4-3d14-4617-a94e-53e71ec20ff6-kube-api-access-g9bbq\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.601467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.601447 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.601524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.601497 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.703008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.702924 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bbq\" (UniqueName: \"kubernetes.io/projected/ea0940e4-3d14-4617-a94e-53e71ec20ff6-kube-api-access-g9bbq\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.703172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.703137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.703239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.703192 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.703952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.703928 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.705534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.705518 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.711049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.711018 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bbq\" (UniqueName: \"kubernetes.io/projected/ea0940e4-3d14-4617-a94e-53e71ec20ff6-kube-api-access-g9bbq\") pod \"maas-api-revoke-sub-b610d2ea-8kddh\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.723713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.723610 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts"] Nov 01 22:38:53.726011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.725980 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:53.726092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.726049 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:53.726092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.726078 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:38:53.842150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.842123 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:53.936522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.936438 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:53.954843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:53.954778 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:54.073340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:54.073305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:54.166467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:54.166442 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh"] Nov 01 22:38:54.441996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:54.441959 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" event={"ID":"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a","Type":"ContainerStarted","Data":"a37210e2dc883f37cf8be844ebd082af064e77e6e0d90e99996b5ab96d2c9db1"} Nov 01 22:38:54.442454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:54.442017 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" event={"ID":"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a","Type":"ContainerStarted","Data":"2901a33b2117f22571f5b4f6d93b981f33e92a41ed9024342ee0e0793f5e6efe"} Nov 01 22:38:54.443871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:54.443837 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" event={"ID":"ea0940e4-3d14-4617-a94e-53e71ec20ff6","Type":"ContainerStarted","Data":"2aa9db796a545f284be72f54640edf27a63909ca4533eebe5cf601c267e9b455"} Nov 01 22:38:54.463180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:54.462414 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:38:54.475567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:54.475486 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podStartSLOduration=1.475468449 podStartE2EDuration="1.475468449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:54.464935146 +0000 UTC m=+899.892813078" watchObservedRunningTime="2026-11-01 22:38:54.475468449 +0000 UTC m=+899.903346379" Nov 01 22:38:55.014077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.014040 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:55.433892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.433858 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:38:55.435361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.435335 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 22:38:55.435485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.435384 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 22:38:55.456060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.456034 4981 generic.go:356] "Generic (PLEG): container finished" podID="ea0940e4-3d14-4617-a94e-53e71ec20ff6" containerID="569ff80db8f626ff0aaabc2ac01cc2eca92a0979add6ad76e18856f18b1c9d43" exitCode=0 Nov 01 22:38:55.456523 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.456210 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" event={"ID":"ea0940e4-3d14-4617-a94e-53e71ec20ff6","Type":"ContainerDied","Data":"569ff80db8f626ff0aaabc2ac01cc2eca92a0979add6ad76e18856f18b1c9d43"} Nov 01 22:38:55.523461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.520888 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9"] Nov 01 22:38:55.528307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.528278 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9"] Nov 01 22:38:55.528460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.528400 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.634312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.634270 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.634488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.634470 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpmv\" (UniqueName: \"kubernetes.io/projected/818e5671-c308-4e66-ad34-5b92925dee1c-kube-api-access-jwpmv\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.634612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.634534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.735992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.735959 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpmv\" (UniqueName: \"kubernetes.io/projected/818e5671-c308-4e66-ad34-5b92925dee1c-kube-api-access-jwpmv\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.736157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.736034 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.736202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.736169 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.736884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.736857 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.738603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.738550 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.744068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.744047 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpmv\" (UniqueName: \"kubernetes.io/projected/818e5671-c308-4e66-ad34-5b92925dee1c-kube-api-access-jwpmv\") pod \"maas-api-revoke-sub-d605a0e0-kths9\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:55.845880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:55.845846 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:56.100024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.099948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:56.159147 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:56.159114 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818e5671_c308_4e66_ad34_5b92925dee1c.slice/crio-6720cee4f478684ea398bef9501705fc9ca696f0e1fbeb45e3dd95c6db62f0f2 WatchSource:0}: Error finding container 6720cee4f478684ea398bef9501705fc9ca696f0e1fbeb45e3dd95c6db62f0f2: Status 404 returned error can't find the container with id 6720cee4f478684ea398bef9501705fc9ca696f0e1fbeb45e3dd95c6db62f0f2 Nov 01 22:38:56.160462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.160435 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9"] Nov 01 22:38:56.234429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.234396 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw"] Nov 01 22:38:56.240550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.240527 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.246621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.246567 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw"] Nov 01 22:38:56.348796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.348751 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.348796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.348801 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9nj\" (UniqueName: \"kubernetes.io/projected/b59b9d86-b77c-4ffd-8926-04043572f32a-kube-api-access-kd9nj\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.349026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.348915 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.435015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.434983 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 22:38:56.435146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.435030 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 22:38:56.450605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.450554 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.450865 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.450845 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.450973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.450888 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9nj\" (UniqueName: \"kubernetes.io/projected/b59b9d86-b77c-4ffd-8926-04043572f32a-kube-api-access-kd9nj\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.451673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.451642 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.452795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.452775 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.459419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.459400 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9nj\" (UniqueName: \"kubernetes.io/projected/b59b9d86-b77c-4ffd-8926-04043572f32a-kube-api-access-kd9nj\") pod \"maas-api-revoke-sub-2d4a5c06-zbzcw\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.464836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.464810 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" event={"ID":"818e5671-c308-4e66-ad34-5b92925dee1c","Type":"ContainerStarted","Data":"6720cee4f478684ea398bef9501705fc9ca696f0e1fbeb45e3dd95c6db62f0f2"} Nov 01 22:38:56.558294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.558245 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:56.743682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.740939 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:56.857609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.857531 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-xks-ca\") pod \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " Nov 01 22:38:56.857783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.857698 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0940e4-3d14-4617-a94e-53e71ec20ff6-kube-api-access-g9bbq\" (UniqueName: \"kubernetes.io/projected/ea0940e4-3d14-4617-a94e-53e71ec20ff6-kube-api-access-g9bbq\") pod \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " Nov 01 22:38:56.858227 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.858020 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-openshift-ca\") pod \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\" (UID: \"ea0940e4-3d14-4617-a94e-53e71ec20ff6\") " Nov 01 22:38:56.858483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.858437 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-openshift-ca" pod "ea0940e4-3d14-4617-a94e-53e71ec20ff6" (UID: "ea0940e4-3d14-4617-a94e-53e71ec20ff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:56.858770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.858749 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:56.860348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.860324 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-xks-ca" pod "ea0940e4-3d14-4617-a94e-53e71ec20ff6" (UID: "ea0940e4-3d14-4617-a94e-53e71ec20ff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:56.861746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.861719 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0940e4-3d14-4617-a94e-53e71ec20ff6-kube-api-access-g9bbq" pod "ea0940e4-3d14-4617-a94e-53e71ec20ff6" (UID: "ea0940e4-3d14-4617-a94e-53e71ec20ff6"). InnerVolumeSpecName "kube-api-access-g9bbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:56.960640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.960563 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0940e4-3d14-4617-a94e-53e71ec20ff6-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:56.960640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:56.960638 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bbq\" (UniqueName: \"kubernetes.io/projected/ea0940e4-3d14-4617-a94e-53e71ec20ff6-kube-api-access-g9bbq\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:57.035735 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:57.035706 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59b9d86_b77c_4ffd_8926_04043572f32a.slice/crio-3fc3088c80a5e0a16e3626c83c5ec9ef2b0da60b80f05ceb2dd35f96202c15d0 WatchSource:0}: Error finding container 3fc3088c80a5e0a16e3626c83c5ec9ef2b0da60b80f05ceb2dd35f96202c15d0: Status 404 returned error can't find the container with id 3fc3088c80a5e0a16e3626c83c5ec9ef2b0da60b80f05ceb2dd35f96202c15d0 Nov 01 22:38:57.036849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.036821 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw"] Nov 01 22:38:57.408633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.407321 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:57.434714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.434683 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 22:38:57.434848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.434738 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 22:38:57.498598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.496127 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" event={"ID":"b59b9d86-b77c-4ffd-8926-04043572f32a","Type":"ContainerStarted","Data":"3fc3088c80a5e0a16e3626c83c5ec9ef2b0da60b80f05ceb2dd35f96202c15d0"} Nov 01 22:38:57.499108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.498722 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" event={"ID":"ea0940e4-3d14-4617-a94e-53e71ec20ff6","Type":"ContainerDied","Data":"2aa9db796a545f284be72f54640edf27a63909ca4533eebe5cf601c267e9b455"} Nov 01 22:38:57.499108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.498769 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aa9db796a545f284be72f54640edf27a63909ca4533eebe5cf601c267e9b455" Nov 01 22:38:57.499108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.498833 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh" Nov 01 22:38:57.501955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.501919 4981 generic.go:356] "Generic (PLEG): container finished" podID="818e5671-c308-4e66-ad34-5b92925dee1c" containerID="3dd5f244aeb02ffe52c5ac4592ac3916a1433fb389a472d0102a493b148455d8" exitCode=0 Nov 01 22:38:57.502083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:57.501975 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" event={"ID":"818e5671-c308-4e66-ad34-5b92925dee1c","Type":"ContainerDied","Data":"3dd5f244aeb02ffe52c5ac4592ac3916a1433fb389a472d0102a493b148455d8"} Nov 01 22:38:58.204019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.203359 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:58.434166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.434133 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 22:38:58.434371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.434180 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 22:38:58.512197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.512163 4981 generic.go:356] "Generic (PLEG): container finished" podID="b59b9d86-b77c-4ffd-8926-04043572f32a" containerID="317a714a610e0bc97c34fdd8134c9c2d47be60eafe20872398ca8d044c1da7c5" exitCode=0 Nov 01 22:38:58.512704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.512247 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" event={"ID":"b59b9d86-b77c-4ffd-8926-04043572f32a","Type":"ContainerDied","Data":"317a714a610e0bc97c34fdd8134c9c2d47be60eafe20872398ca8d044c1da7c5"} Nov 01 22:38:58.746494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.746464 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:58.818443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.818411 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp"] Nov 01 22:38:58.819642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.819618 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-openshift-ca\") pod \"818e5671-c308-4e66-ad34-5b92925dee1c\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " Nov 01 22:38:58.819892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.819697 4981 cpu_manager.go:414] "containerMap: removing container" podUID="818e5671-c308-4e66-ad34-5b92925dee1c" containerName="revoke-keys" Nov 01 22:38:58.820019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.820003 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818e5671-c308-4e66-ad34-5b92925dee1c" containerName="revoke-keys" Nov 01 22:38:58.820205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.820191 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ea0940e4-3d14-4617-a94e-53e71ec20ff6" containerName="revoke-keys" Nov 01 22:38:58.820324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.820311 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0940e4-3d14-4617-a94e-53e71ec20ff6" containerName="revoke-keys" Nov 01 22:38:58.820470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.820048 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-openshift-ca" pod "818e5671-c308-4e66-ad34-5b92925dee1c" (UID: "818e5671-c308-4e66-ad34-5b92925dee1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:58.820600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.820571 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0940e4-3d14-4617-a94e-53e71ec20ff6" containerName="revoke-keys" Nov 01 22:38:58.820724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.820710 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="818e5671-c308-4e66-ad34-5b92925dee1c" containerName="revoke-keys" Nov 01 22:38:58.820811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.819939 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818e5671-c308-4e66-ad34-5b92925dee1c-kube-api-access-jwpmv\" (UniqueName: \"kubernetes.io/projected/818e5671-c308-4e66-ad34-5b92925dee1c-kube-api-access-jwpmv\") pod \"818e5671-c308-4e66-ad34-5b92925dee1c\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " Nov 01 22:38:58.820941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.820926 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-xks-ca\") pod \"818e5671-c308-4e66-ad34-5b92925dee1c\" (UID: \"818e5671-c308-4e66-ad34-5b92925dee1c\") " Nov 01 22:38:58.821919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.821896 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:58.822657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.822589 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818e5671-c308-4e66-ad34-5b92925dee1c-kube-api-access-jwpmv" pod "818e5671-c308-4e66-ad34-5b92925dee1c" (UID: "818e5671-c308-4e66-ad34-5b92925dee1c"). InnerVolumeSpecName "kube-api-access-jwpmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:58.825789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.825762 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-xks-ca" pod "818e5671-c308-4e66-ad34-5b92925dee1c" (UID: "818e5671-c308-4e66-ad34-5b92925dee1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:58.826158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.826134 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:58.830490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.830463 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp"] Nov 01 22:38:58.923737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.923699 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdj7b\" (UniqueName: \"kubernetes.io/projected/29207512-8f8c-4067-849a-6fd359b2185a-kube-api-access-mdj7b\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:58.923903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.923853 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:58.923981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.923958 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:58.924171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.924159 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwpmv\" (UniqueName: \"kubernetes.io/projected/818e5671-c308-4e66-ad34-5b92925dee1c-kube-api-access-jwpmv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:58.924204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:58.924175 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818e5671-c308-4e66-ad34-5b92925dee1c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:59.025157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.025061 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdj7b\" (UniqueName: \"kubernetes.io/projected/29207512-8f8c-4067-849a-6fd359b2185a-kube-api-access-mdj7b\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:59.025157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.025154 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:59.025344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.025207 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:59.025815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.025791 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:59.027611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.027585 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:59.038570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.038550 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdj7b\" (UniqueName: \"kubernetes.io/projected/29207512-8f8c-4067-849a-6fd359b2185a-kube-api-access-mdj7b\") pod \"maas-api-revoke-sub-80ef5042-qwcdp\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:59.201257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.201221 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:38:59.251509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.251481 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:38:59.434792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.434756 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 22:38:59.434963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.434813 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 22:38:59.503891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.503812 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp"] Nov 01 22:38:59.503986 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:38:59.503937 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29207512_8f8c_4067_849a_6fd359b2185a.slice/crio-fc7627acc0abff88dc2935705b993791125c6edbfc744f273243cf0a0b6585fd WatchSource:0}: Error finding container fc7627acc0abff88dc2935705b993791125c6edbfc744f273243cf0a0b6585fd: Status 404 returned error can't find the container with id fc7627acc0abff88dc2935705b993791125c6edbfc744f273243cf0a0b6585fd Nov 01 22:38:59.522553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.522305 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" event={"ID":"29207512-8f8c-4067-849a-6fd359b2185a","Type":"ContainerStarted","Data":"fc7627acc0abff88dc2935705b993791125c6edbfc744f273243cf0a0b6585fd"} Nov 01 22:38:59.523747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.523732 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" Nov 01 22:38:59.523811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.523778 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9" event={"ID":"818e5671-c308-4e66-ad34-5b92925dee1c","Type":"ContainerDied","Data":"6720cee4f478684ea398bef9501705fc9ca696f0e1fbeb45e3dd95c6db62f0f2"} Nov 01 22:38:59.523844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.523834 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6720cee4f478684ea398bef9501705fc9ca696f0e1fbeb45e3dd95c6db62f0f2" Nov 01 22:38:59.747721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.747689 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:38:59.837095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.837052 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-openshift-ca\") pod \"b59b9d86-b77c-4ffd-8926-04043572f32a\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " Nov 01 22:38:59.837242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.837127 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59b9d86-b77c-4ffd-8926-04043572f32a-kube-api-access-kd9nj\" (UniqueName: \"kubernetes.io/projected/b59b9d86-b77c-4ffd-8926-04043572f32a-kube-api-access-kd9nj\") pod \"b59b9d86-b77c-4ffd-8926-04043572f32a\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " Nov 01 22:38:59.837242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.837179 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-xks-ca\") pod \"b59b9d86-b77c-4ffd-8926-04043572f32a\" (UID: \"b59b9d86-b77c-4ffd-8926-04043572f32a\") " Nov 01 22:38:59.838020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.837990 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-openshift-ca" pod "b59b9d86-b77c-4ffd-8926-04043572f32a" (UID: "b59b9d86-b77c-4ffd-8926-04043572f32a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:38:59.839889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.839864 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59b9d86-b77c-4ffd-8926-04043572f32a-kube-api-access-kd9nj" pod "b59b9d86-b77c-4ffd-8926-04043572f32a" (UID: "b59b9d86-b77c-4ffd-8926-04043572f32a"). InnerVolumeSpecName "kube-api-access-kd9nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:38:59.839990 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.839960 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-xks-ca" pod "b59b9d86-b77c-4ffd-8926-04043572f32a" (UID: "b59b9d86-b77c-4ffd-8926-04043572f32a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:38:59.939524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.939484 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:59.939524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.939517 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd9nj\" (UniqueName: \"kubernetes.io/projected/b59b9d86-b77c-4ffd-8926-04043572f32a-kube-api-access-kd9nj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:38:59.939524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:38:59.939528 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b9d86-b77c-4ffd-8926-04043572f32a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:00.242551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.242477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:00.434770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.434729 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 01 22:39:00.434940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.434784 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 01 22:39:00.532549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.532461 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" Nov 01 22:39:00.533072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.532472 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw" event={"ID":"b59b9d86-b77c-4ffd-8926-04043572f32a","Type":"ContainerDied","Data":"3fc3088c80a5e0a16e3626c83c5ec9ef2b0da60b80f05ceb2dd35f96202c15d0"} Nov 01 22:39:00.533072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.532607 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc3088c80a5e0a16e3626c83c5ec9ef2b0da60b80f05ceb2dd35f96202c15d0" Nov 01 22:39:00.534351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.534321 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" event={"ID":"29207512-8f8c-4067-849a-6fd359b2185a","Type":"ContainerStarted","Data":"14b2986848df9e93a99449386dfaccdbf927650927d089e9e931218fc2485145"} Nov 01 22:39:00.559712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.559660 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" podStartSLOduration=2.559643073 podStartE2EDuration="2.559643073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:00.55231104 +0000 UTC m=+905.980188973" watchObservedRunningTime="2026-11-01 22:39:00.559643073 +0000 UTC m=+905.987521005" Nov 01 22:39:00.958172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:00.958142 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:01.438922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:01.438887 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:39:01.439286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:01.439260 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:39:01.440040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:01.440020 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:39:01.545298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:01.545259 4981 generic.go:356] "Generic (PLEG): container finished" podID="29207512-8f8c-4067-849a-6fd359b2185a" containerID="14b2986848df9e93a99449386dfaccdbf927650927d089e9e931218fc2485145" exitCode=0 Nov 01 22:39:01.545832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:01.545339 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" event={"ID":"29207512-8f8c-4067-849a-6fd359b2185a","Type":"ContainerDied","Data":"14b2986848df9e93a99449386dfaccdbf927650927d089e9e931218fc2485145"} Nov 01 22:39:01.767290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:01.767200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:02.475003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.474962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:02.745365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.745280 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:39:02.885500 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.885460 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29207512-8f8c-4067-849a-6fd359b2185a-kube-api-access-mdj7b\" (UniqueName: \"kubernetes.io/projected/29207512-8f8c-4067-849a-6fd359b2185a-kube-api-access-mdj7b\") pod \"29207512-8f8c-4067-849a-6fd359b2185a\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " Nov 01 22:39:02.885812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.885778 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-xks-ca\") pod \"29207512-8f8c-4067-849a-6fd359b2185a\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " Nov 01 22:39:02.885908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.885817 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-openshift-ca\") pod \"29207512-8f8c-4067-849a-6fd359b2185a\" (UID: \"29207512-8f8c-4067-849a-6fd359b2185a\") " Nov 01 22:39:02.886318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.886257 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-openshift-ca" pod "29207512-8f8c-4067-849a-6fd359b2185a" (UID: "29207512-8f8c-4067-849a-6fd359b2185a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:02.886675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.886654 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:02.888127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.888100 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-xks-ca" pod "29207512-8f8c-4067-849a-6fd359b2185a" (UID: "29207512-8f8c-4067-849a-6fd359b2185a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:02.888858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.888833 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29207512-8f8c-4067-849a-6fd359b2185a-kube-api-access-mdj7b" pod "29207512-8f8c-4067-849a-6fd359b2185a" (UID: "29207512-8f8c-4067-849a-6fd359b2185a"). InnerVolumeSpecName "kube-api-access-mdj7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:02.987902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.987869 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29207512-8f8c-4067-849a-6fd359b2185a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:02.987902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:02.987898 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdj7b\" (UniqueName: \"kubernetes.io/projected/29207512-8f8c-4067-849a-6fd359b2185a-kube-api-access-mdj7b\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:03.220961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:03.220921 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:03.564382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:03.564301 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" Nov 01 22:39:03.564520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:03.564297 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp" event={"ID":"29207512-8f8c-4067-849a-6fd359b2185a","Type":"ContainerDied","Data":"fc7627acc0abff88dc2935705b993791125c6edbfc744f273243cf0a0b6585fd"} Nov 01 22:39:03.564520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:03.564428 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc7627acc0abff88dc2935705b993791125c6edbfc744f273243cf0a0b6585fd" Nov 01 22:39:03.866508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:03.866428 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:04.553814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:04.553781 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:05.357566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:05.357530 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:06.244885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:06.244854 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:06.954974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:06.954944 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:07.840703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:07.840671 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:08.116292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.116208 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6"] Nov 01 22:39:08.117370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.117345 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b59b9d86-b77c-4ffd-8926-04043572f32a" containerName="revoke-keys" Nov 01 22:39:08.117370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.117370 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59b9d86-b77c-4ffd-8926-04043572f32a" containerName="revoke-keys" Nov 01 22:39:08.117554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.117405 4981 cpu_manager.go:414] "containerMap: removing container" podUID="29207512-8f8c-4067-849a-6fd359b2185a" containerName="revoke-keys" Nov 01 22:39:08.117554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.117413 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29207512-8f8c-4067-849a-6fd359b2185a" containerName="revoke-keys" Nov 01 22:39:08.117716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.117624 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59b9d86-b77c-4ffd-8926-04043572f32a" containerName="revoke-keys" Nov 01 22:39:08.117716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.117648 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="29207512-8f8c-4067-849a-6fd359b2185a" containerName="revoke-keys" Nov 01 22:39:08.122321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.122298 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.160873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.160832 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6"] Nov 01 22:39:08.169175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.169146 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z"] Nov 01 22:39:08.175074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.175049 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.176122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.176098 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.176305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.176285 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.176471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.176454 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.176638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.176619 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.177622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.176859 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.177622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.177097 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.177622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.177138 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4n6r\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-kube-api-access-k4n6r\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.177622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.177185 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.177622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.177241 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.183016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.182990 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z"] Nov 01 22:39:08.278996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.278965 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279019 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279043 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279066 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279100 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.279354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279226 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c72bed6-36e0-47d4-b761-53d752aa7962-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.279354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279285 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279334 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxbn\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-kube-api-access-qxxbn\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.279467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279372 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279455 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279478 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.279588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279528 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279548 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279631 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279679 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.279768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279706 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.279911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279803 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.279911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279816 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.279999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.279950 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.280035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.280022 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.280079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.280042 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.280079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.280072 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4n6r\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-kube-api-access-k4n6r\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.280726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.280699 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.281535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.281518 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.282065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.282049 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.287265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.287244 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.287932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.287914 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4n6r\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-kube-api-access-k4n6r\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.381153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381081 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381153 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381242 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381265 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c72bed6-36e0-47d4-b761-53d752aa7962-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381300 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxbn\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-kube-api-access-qxxbn\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381347 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381390 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381418 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381478 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381633 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.381869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.381842 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-data\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.382046 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.382027 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-credential-socket\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.382096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.382065 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-certs\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.382201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.382182 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c72bed6-36e0-47d4-b761-53d752aa7962-istiod-ca-cert\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.383943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.383917 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-envoy\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.384179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.384159 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-podinfo\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.392317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.392296 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-token\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.392664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.392647 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxbn\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-kube-api-access-qxxbn\") pod \"e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.457023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.456990 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:08.494373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.494330 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:08.632961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.632886 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm"] Nov 01 22:39:08.640047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.640009 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.652437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.652409 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm"] Nov 01 22:39:08.660690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.660652 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg"] Nov 01 22:39:08.673716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.672263 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg"] Nov 01 22:39:08.673716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.672436 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.685720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.685691 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.685858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.685844 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.685963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.685949 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.686006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.685974 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.686050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.686038 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.686086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.686064 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.686119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.686089 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgfjs\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-kube-api-access-zgfjs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.686119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.686106 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.686258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.686243 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.748965 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:08.748934 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7f3fdd_d5d9_4c31_b98d_e37188a72983.slice/crio-8b984ecb5ab7c38d9eaf3b2381e07af4784d5189686d56da7e95f39212376d4e WatchSource:0}: Error finding container 8b984ecb5ab7c38d9eaf3b2381e07af4784d5189686d56da7e95f39212376d4e: Status 404 returned error can't find the container with id 8b984ecb5ab7c38d9eaf3b2381e07af4784d5189686d56da7e95f39212376d4e Nov 01 22:39:08.750291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.750267 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6"] Nov 01 22:39:08.751387 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.751357 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:08.751512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.751418 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:08.751512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.751444 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:08.787685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.787658 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.787805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.787715 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.787866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.787843 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.788027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788000 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.788098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788040 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.788098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788046 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.788364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788095 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.788364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788152 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.788364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788227 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.788364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788260 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.788364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788285 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgfjs\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-kube-api-access-zgfjs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.788364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788312 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.788364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788313 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788429 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788446 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788519 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788592 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788651 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788714 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788742 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfgw\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-kube-api-access-9hfgw\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788795 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.788863 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.789780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.789372 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.790804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.790778 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.791674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.791652 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.810134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.810103 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.810254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.810238 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgfjs\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-kube-api-access-zgfjs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.824392 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:08.824343 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c72bed6_36e0_47d4_b761_53d752aa7962.slice/crio-40b720219c32bd2f028e71010d2cef14074dacf7d593c3265e51effcd9d27127 WatchSource:0}: Error finding container 40b720219c32bd2f028e71010d2cef14074dacf7d593c3265e51effcd9d27127: Status 404 returned error can't find the container with id 40b720219c32bd2f028e71010d2cef14074dacf7d593c3265e51effcd9d27127 Nov 01 22:39:08.826472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.826422 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z"] Nov 01 22:39:08.827944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.827913 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:08.828031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.827983 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:08.828074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.828037 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:08.891094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.890992 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891184 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891258 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfgw\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-kube-api-access-9hfgw\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891307 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891370 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891424 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891501 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891657 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.891734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.891710 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.893975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.892708 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-data\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.893975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.893244 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istiod-ca-cert\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.893975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.893528 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-credential-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.893975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.893808 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-certs\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.894237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.894014 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-socket\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.895186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.895164 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-envoy\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.908620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.907113 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-podinfo\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.914631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.914537 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-token\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.918023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.917997 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfgw\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-kube-api-access-9hfgw\") pod \"e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:08.938605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.931446 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:08.969655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.967838 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:08.990476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:08.990241 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:09.318334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.318304 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg"] Nov 01 22:39:09.333655 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:09.326121 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede5d133_8820_48a7_8b9b_6c6b4fef23ff.slice/crio-d900504021a407fc17e2ad1e4ad907680fda52e6f1bbe777739f774dc3c25b46 WatchSource:0}: Error finding container d900504021a407fc17e2ad1e4ad907680fda52e6f1bbe777739f774dc3c25b46: Status 404 returned error can't find the container with id d900504021a407fc17e2ad1e4ad907680fda52e6f1bbe777739f774dc3c25b46 Nov 01 22:39:09.335695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.335609 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:09.335695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.335688 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:09.335877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.335730 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:09.336427 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:09.336290 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9775d38d_d819_4c0d_a5a4_5c006a9c2511.slice/crio-c977872441fa8ee833483e0870122dcbb5df1a4936b3ddc690163ccbe6da2744 WatchSource:0}: Error finding container c977872441fa8ee833483e0870122dcbb5df1a4936b3ddc690163ccbe6da2744: Status 404 returned error can't find the container with id c977872441fa8ee833483e0870122dcbb5df1a4936b3ddc690163ccbe6da2744 Nov 01 22:39:09.336945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.336920 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm"] Nov 01 22:39:09.341264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.341231 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:09.341337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.341309 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:09.341390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.341347 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:09.623980 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.623940 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" event={"ID":"fb7f3fdd-d5d9-4c31-b98d-e37188a72983","Type":"ContainerStarted","Data":"39f53f2729c7afb2a223021496a5dbb442762bddcd662ed0b8d9c12a04b5c8f7"} Nov 01 22:39:09.624185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.624007 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" event={"ID":"fb7f3fdd-d5d9-4c31-b98d-e37188a72983","Type":"ContainerStarted","Data":"8b984ecb5ab7c38d9eaf3b2381e07af4784d5189686d56da7e95f39212376d4e"} Nov 01 22:39:09.629706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.629662 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" event={"ID":"9775d38d-d819-4c0d-a5a4-5c006a9c2511","Type":"ContainerStarted","Data":"2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc"} Nov 01 22:39:09.634419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.629946 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" event={"ID":"9775d38d-d819-4c0d-a5a4-5c006a9c2511","Type":"ContainerStarted","Data":"c977872441fa8ee833483e0870122dcbb5df1a4936b3ddc690163ccbe6da2744"} Nov 01 22:39:09.635852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.635815 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" event={"ID":"1c72bed6-36e0-47d4-b761-53d752aa7962","Type":"ContainerStarted","Data":"39b862b8137fde23d327c3f5aa77486ecec1b56deb3834e32eea43b5dfc30aab"} Nov 01 22:39:09.635952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.635862 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" event={"ID":"1c72bed6-36e0-47d4-b761-53d752aa7962","Type":"ContainerStarted","Data":"40b720219c32bd2f028e71010d2cef14074dacf7d593c3265e51effcd9d27127"} Nov 01 22:39:09.638636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.638566 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" event={"ID":"ede5d133-8820-48a7-8b9b-6c6b4fef23ff","Type":"ContainerStarted","Data":"468e13ea71f93d4c825adff3df1cc942e10b8ef9bb4bf9a849ed7d8459552eac"} Nov 01 22:39:09.638752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.638661 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" event={"ID":"ede5d133-8820-48a7-8b9b-6c6b4fef23ff","Type":"ContainerStarted","Data":"d900504021a407fc17e2ad1e4ad907680fda52e6f1bbe777739f774dc3c25b46"} Nov 01 22:39:09.659376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.658800 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podStartSLOduration=1.65878574 podStartE2EDuration="1.65878574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.64880721 +0000 UTC m=+915.076685142" watchObservedRunningTime="2026-11-01 22:39:09.65878574 +0000 UTC m=+915.086663671" Nov 01 22:39:09.666361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.666326 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp"] Nov 01 22:39:09.673766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.673708 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp"] Nov 01 22:39:09.673943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.673866 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.679997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.678615 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:39:09.679997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.678703 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:39:09.679997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.678788 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:39:09.680281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.680134 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podStartSLOduration=1.680118342 podStartE2EDuration="1.680118342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.677467276 +0000 UTC m=+915.105345210" watchObservedRunningTime="2026-11-01 22:39:09.680118342 +0000 UTC m=+915.107996274" Nov 01 22:39:09.694204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.694178 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:39:09.731336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.731277 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podStartSLOduration=1.731257965 podStartE2EDuration="1.731257965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.726360791 +0000 UTC m=+915.154238722" watchObservedRunningTime="2026-11-01 22:39:09.731257965 +0000 UTC m=+915.159135896" Nov 01 22:39:09.736356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.731353 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.736798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.736769 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd955\" (UniqueName: \"kubernetes.io/projected/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-kube-api-access-qd955\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.739530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.738557 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.756619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.752871 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podStartSLOduration=1.752853257 podStartE2EDuration="1.752853257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:09.751338978 +0000 UTC m=+915.179216910" watchObservedRunningTime="2026-11-01 22:39:09.752853257 +0000 UTC m=+915.180731184" Nov 01 22:39:09.845558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.845509 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd955\" (UniqueName: \"kubernetes.io/projected/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-kube-api-access-qd955\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.846117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.846094 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.846346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.846328 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.846734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.846712 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.849033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.849012 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.863152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.863126 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd955\" (UniqueName: \"kubernetes.io/projected/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-kube-api-access-qd955\") pod \"maas-api-revoke-sub-58f854ec-td7kp\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:09.968353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.968269 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:09.991708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:09.991664 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:10.014893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:10.014854 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:10.346596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:10.346493 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:10.553001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:10.505399 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:10.553001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:10.516778 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:10.553001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:10.544567 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:10.553001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:10.544644 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:11.042639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.034801 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.042639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.034869 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:11.136063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.104432 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.136063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.104495 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:11.186867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.186826 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.317165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.315716 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.404657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.330735 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.404657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.330795 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:11.404657 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:11.331845 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdfd5b3_f010_40ef_91c0_7cb5a16e801a.slice/crio-70b4f8ab2c6b8ba837f13fbdf82b7060ac502776cc779c06fca0678e2f858ff5 WatchSource:0}: Error finding container 70b4f8ab2c6b8ba837f13fbdf82b7060ac502776cc779c06fca0678e2f858ff5: Status 404 returned error can't find the container with id 70b4f8ab2c6b8ba837f13fbdf82b7060ac502776cc779c06fca0678e2f858ff5 Nov 01 22:39:11.404657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.337654 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:11.404657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.337940 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:11.471617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.464038 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.471617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.464095 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:11.471617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.464165 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.471617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.464219 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.510047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.509937 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.538647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.520177 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.538647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.520230 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:11.562042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.561922 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.564558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.564519 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.579631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.574645 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:11.666912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.666799 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:11.774062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.684800 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp"] Nov 01 22:39:11.774062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.684867 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2"] Nov 01 22:39:11.778794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.778751 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:11.802950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.802547 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:11.805407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.805379 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2"] Nov 01 22:39:11.823869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.805671 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:11.968603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.957895 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhccb\" (UniqueName: \"kubernetes.io/projected/35583d35-83a2-4e3d-9997-6773a898f70f-kube-api-access-mhccb\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:11.968603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.958102 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:11.968603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.958131 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:11.980317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.980272 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:11.980599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:11.980548 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:12.007107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.007073 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.013495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.013463 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:12.091740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.075822 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhccb\" (UniqueName: \"kubernetes.io/projected/35583d35-83a2-4e3d-9997-6773a898f70f-kube-api-access-mhccb\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:12.091740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.076296 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:12.091740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.076329 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:12.093225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.093189 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:12.152602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.149047 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:12.152602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.149466 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:12.190361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.190321 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:12.202952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.197254 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:12.203192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.203037 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:39:12.340054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.340000 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.340702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.340654 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.352856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.352748 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:12.376839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.365708 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:39:12.395911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.395866 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:39:12.396192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.396175 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:39:12.471585 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.471524 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.475057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.471467 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.475673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.475646 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:39:12.496803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.496761 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:39:12.524601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.473437 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.525162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.524956 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:12.525358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.522023 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:12.526023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.499041 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:12.526204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.526179 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:12.526422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.526406 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:39:12.569626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.562637 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:12.569626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.563050 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:39:12.608316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.608202 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:12.608717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.608692 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:39:12.659324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.610026 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:12.659324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.610348 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:39:12.659324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.638254 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhccb\" (UniqueName: \"kubernetes.io/projected/35583d35-83a2-4e3d-9997-6773a898f70f-kube-api-access-mhccb\") pod \"maas-api-revoke-sub-bb55705d-bpcv2\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:12.670629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.670539 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:12.671050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.671021 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:39:12.721609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.719034 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" event={"ID":"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a","Type":"ContainerStarted","Data":"70b4f8ab2c6b8ba837f13fbdf82b7060ac502776cc779c06fca0678e2f858ff5"} Nov 01 22:39:12.818727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.810109 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:12.885750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.885642 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:12.901602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:12.886642 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:13.164658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.150776 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.193600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.166027 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.283657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.228162 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.283657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.228225 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:13.283657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.253368 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:39:13.283657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.253434 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:39:13.283657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.253493 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:39:13.283657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.253530 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:39:13.284320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.284260 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.286258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.285597 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.286480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.286445 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:13.286661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.286621 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.364915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.364861 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:13.372849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.372813 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:13.400801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.400763 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.616600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.597717 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:13.616600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.598167 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:39:13.645850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.645800 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.650934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.650891 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.694232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.694187 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:13.791024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.790976 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:13.958747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.958697 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.959077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.959045 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:13.966857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.966815 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:13.967098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:13.967060 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:14.017895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.017840 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.324703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.116029 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:14.324703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.152933 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.499598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.499056 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.522460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.500987 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:14.606979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.606937 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:39:14.644073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.644026 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.702600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.684863 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.757598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.703321 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.757598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.739100 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.782657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.767764 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.790714 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.791987 4981 patch_prober.go:29] interesting pod/jobset-controller-manager-656887d9d4-psjhj container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.792030 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-656887d9d4-psjhj" podUID="b3464466-11fd-4690-bd6a-7d6a80854fac" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.799226 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.799267 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.814750 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.814798 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.827676 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:14.841601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.829493 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.880160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.845310 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:14.880160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.845364 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:14.880160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.862296 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:14.972645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.959787 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:15.075659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:14.981756 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:15.075659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.040790 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.120078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.078919 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.120078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.078975 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:15.120078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.079047 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.120078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.079070 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:15.120078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.079126 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.120078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.079147 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:15.405832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.405719 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.433603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.411017 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:15.536684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.536630 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:15.605826 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.605789 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.606103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.606074 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:15.606277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.606260 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:15.606405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.606385 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:15.609147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.609118 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.609667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.609644 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:15.629832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.629795 4981 generic.go:356] "Generic (PLEG): container finished" podID="9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" containerID="584d560864863bcaf19adc7a91ad6405e2f8ca36c5632a31d139114981f3fbac" exitCode=0 Nov 01 22:39:15.641940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.636507 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:39:15.764684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.687452 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:15.764684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.712690 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:15.764684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.743645 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.764684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.743761 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:15.785769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.785728 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.916602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.804271 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.924735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.877206 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.925010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.877603 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:15.925161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.894682 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.960612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.958215 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 01 22:39:15.985627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.961430 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.985981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.985945 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:15.986198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.986164 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.008588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:15.687374 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.413987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.413866 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:16.517614 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:39:16.471297 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.585s" Nov 01 22:39:16.517614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.471339 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" event={"ID":"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a","Type":"ContainerDied","Data":"584d560864863bcaf19adc7a91ad6405e2f8ca36c5632a31d139114981f3fbac"} Nov 01 22:39:16.517614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.503717 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.544309 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.544371 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.558767 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.558832 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.559282 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.559502 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.559537 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.567866 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.567906 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:16.571598 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:16.569932 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35583d35_83a2_4e3d_9997_6773a898f70f.slice/crio-48a70b02dbfa0df1f1ab1ee2ad1de0f413068344c345b5e35448147549d42ba5 WatchSource:0}: Error finding container 48a70b02dbfa0df1f1ab1ee2ad1de0f413068344c345b5e35448147549d42ba5: Status 404 returned error can't find the container with id 48a70b02dbfa0df1f1ab1ee2ad1de0f413068344c345b5e35448147549d42ba5 Nov 01 22:39:16.616675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.610524 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.616675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.611901 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:16.641961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:16.628294 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:16.917659 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:39:16.893532 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:39:17.035912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.035702 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.035912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.035763 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:17.057602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.037278 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.057602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.037333 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:17.462680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.458998 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:17.491036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.488879 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:17.492128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.492099 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.492307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.492282 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:17.492487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.492467 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.524570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.524434 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.533615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.525210 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.533615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.526989 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:17.533615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.528048 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:17.533615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.524674 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:17.534064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.524905 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:39:17.746959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.524693 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:17.746959 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:39:17.743282 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.013s" Nov 01 22:39:17.746959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.743462 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" event={"ID":"35583d35-83a2-4e3d-9997-6773a898f70f","Type":"ContainerStarted","Data":"48a70b02dbfa0df1f1ab1ee2ad1de0f413068344c345b5e35448147549d42ba5"} Nov 01 22:39:17.746959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.743535 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2"] Nov 01 22:39:17.746959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.524646 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.781648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.524734 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:17.794006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.537714 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.574079 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.820904 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.601936 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.821007 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.717672 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.821068 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.729198 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.821113 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.610884 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.829629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.613647 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:17.975984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:17.524711 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:18.003427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.002822 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.003427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.002897 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:18.044886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.004307 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Nov 01 22:39:18.044886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.005640 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.044886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.005682 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:18.059040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.058999 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:18.059299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.059268 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.215633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.202739 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:18.215633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.202848 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.483666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.483229 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.500131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.500087 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.500773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.500743 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:18.529962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.529907 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.535050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.531186 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.538921 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.538892 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:18.567807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.567764 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:18.694785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.694741 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.695082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.695051 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:18.695234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.694873 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:18.695349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.695325 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:18.920604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.913619 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.097659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.953023 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:19.188599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.953073 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:19.189158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.953093 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:19.189554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.953114 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.189958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:18.953852 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.206385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.008884 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.206621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.206570 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:19.206771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.043490 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:19.207308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.042710 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.207669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.168623 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.207824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.207802 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:19.371744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.348886 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.555773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.542367 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:39:19.583392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.583339 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.584121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.584074 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.584562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.584539 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.585173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.585150 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.591537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.590726 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.591537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.591202 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.610677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.610633 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:19.638412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.638325 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.639600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.639549 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:19.639853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.639833 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:19.656091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.656049 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:19.912735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.912653 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:19.980033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:19.980001 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.033720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.029828 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:20.033720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.030302 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.033720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.030365 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:20.049171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.049141 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-openshift-ca\") pod \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " Nov 01 22:39:20.057409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.057376 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-xks-ca\") pod \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " Nov 01 22:39:20.062869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.062840 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-kube-api-access-qd955\" (UniqueName: \"kubernetes.io/projected/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-kube-api-access-qd955\") pod \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\" (UID: \"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a\") " Nov 01 22:39:20.079881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.056766 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-openshift-ca" pod "9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" (UID: "9cdfd5b3-f010-40ef-91c0-7cb5a16e801a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:20.099516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.099470 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.258735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.257762 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.274600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.259049 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:20.274600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.259233 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:20.274600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.259417 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:20.274600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.259594 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:20.274600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.259760 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:20.274600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.259960 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:20.274600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.259994 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.407925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.407895 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:20.417423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.417391 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-xks-ca" pod "9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" (UID: "9cdfd5b3-f010-40ef-91c0-7cb5a16e801a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:20.430643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.411767 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:20.430918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.430879 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.495604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.466784 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.506472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.483742 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.506716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.506688 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:20.506842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.484277 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.516684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.508699 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:20.516684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.508749 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:20.516684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.511036 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:20.596600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.586126 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:20.596600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.589672 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.601008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.589708 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:20.601008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.589761 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.605731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.604674 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:20.610222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.609651 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:20.640683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.640052 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:20.752607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.751176 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:20.752607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.751307 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr"] Nov 01 22:39:20.752607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.752474 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" containerName="revoke-keys" Nov 01 22:39:20.752607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.752495 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" containerName="revoke-keys" Nov 01 22:39:20.752942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.752757 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" containerName="revoke-keys" Nov 01 22:39:20.817737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.811243 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:20.933600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.909408 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-kube-api-access-qd955" pod "9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" (UID: "9cdfd5b3-f010-40ef-91c0-7cb5a16e801a"). InnerVolumeSpecName "kube-api-access-qd955". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.003946 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.040757 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:20.985987 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.040894 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmbr\" (UniqueName: \"kubernetes.io/projected/aad69459-832e-4a63-b7a3-3fb5cd482c4b-kube-api-access-mhmbr\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.040948 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.041210 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd955\" (UniqueName: \"kubernetes.io/projected/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a-kube-api-access-qd955\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.013447 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.049606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.041281 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:21.078185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.076975 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" event={"ID":"9cdfd5b3-f010-40ef-91c0-7cb5a16e801a","Type":"ContainerDied","Data":"70b4f8ab2c6b8ba837f13fbdf82b7060ac502776cc779c06fca0678e2f858ff5"} Nov 01 22:39:21.078185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.077030 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70b4f8ab2c6b8ba837f13fbdf82b7060ac502776cc779c06fca0678e2f858ff5" Nov 01 22:39:21.080464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.079752 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp" Nov 01 22:39:21.093537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.082748 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" event={"ID":"35583d35-83a2-4e3d-9997-6773a898f70f","Type":"ContainerStarted","Data":"c5d6239fe24f368e7f9bdff9267066dbd16c9fff581a920a96feda2fa191c58c"} Nov 01 22:39:21.108512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.108482 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr"] Nov 01 22:39:21.168602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.142346 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.370687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.359821 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmbr\" (UniqueName: \"kubernetes.io/projected/aad69459-832e-4a63-b7a3-3fb5cd482c4b-kube-api-access-mhmbr\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.371010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.370977 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.447603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.357725 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:39:21.610601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.409898 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:21.611175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.409934 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:21.611670 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.409976 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:21.615596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.409999 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:21.619595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.411350 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.619748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.357768 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.418009 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.418942 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.463636 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmbr\" (UniqueName: \"kubernetes.io/projected/aad69459-832e-4a63-b7a3-3fb5cd482c4b-kube-api-access-mhmbr\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.479521 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.654875 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.604992 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.508463 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.501203 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.659114 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.607833 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc63a4d1-2c2nr\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.619753 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.605058 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.605085 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.662649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.660050 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:39:21.730600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.640405 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.732312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.605119 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:21.782598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.634283 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:21.782598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.761285 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:22.023601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.997196 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.048601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.024106 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:22.238628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:21.998101 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.239182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.239146 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:22.473676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.463949 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.473676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.464010 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:22.535656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.499196 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:22.535656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.499251 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:22.595807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.592744 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.618689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.616742 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:22.618689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.617493 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:22.618689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.617791 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:22.618689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.618029 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:22.633776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.633719 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" podStartSLOduration=12.633702276 podStartE2EDuration="12.633702276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:21.11671577 +0000 UTC m=+926.544593702" watchObservedRunningTime="2026-11-01 22:39:22.633702276 +0000 UTC m=+928.061580208" Nov 01 22:39:22.646480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.646420 4981 generic.go:356] "Generic (PLEG): container finished" podID="35583d35-83a2-4e3d-9997-6773a898f70f" containerID="c5d6239fe24f368e7f9bdff9267066dbd16c9fff581a920a96feda2fa191c58c" exitCode=0 Nov 01 22:39:22.657742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.657707 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" event={"ID":"35583d35-83a2-4e3d-9997-6773a898f70f","Type":"ContainerDied","Data":"c5d6239fe24f368e7f9bdff9267066dbd16c9fff581a920a96feda2fa191c58c"} Nov 01 22:39:22.674756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.658766 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.676180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.661219 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:22.702302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.661265 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:22.702753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.665277 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.721827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.665304 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:22.722327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.665323 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:22.722725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.665342 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:22.826655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.748269 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:22.826655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.780382 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:22.827136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:22.827096 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.129657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.031231 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.129657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.031291 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:23.129657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.046356 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.129657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.046408 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:23.305705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.305658 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.466468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.460330 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:23.468075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.467810 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.468075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.467859 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:23.534638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.499992 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:23.534638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.500046 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:23.534638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.500527 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:23.542413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.542376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:23.628649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.615189 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.661732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.628915 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.667140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.628915 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.673606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.628948 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:23.673606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.628981 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.694760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.694716 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.712704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.712655 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:23.713812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.713780 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:23.718112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.718046 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:23.734891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.734849 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.736629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.736601 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:23.737386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.737361 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:23.749512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.749480 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:23.786098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.785263 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.906594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.891924 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:23.926595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.924835 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.007060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.985013 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:24.007060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.994190 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.007060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.994236 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 22:39:24.007060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:23.998699 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:24.007060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.006473 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.007060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.006521 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 01 22:39:24.084939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.081944 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:39:24.165767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.163219 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:39:24.480176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.480133 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.580659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.480873 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:39:24.580659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.576292 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:39:24.580659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.547667 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.580659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.577502 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 01 22:39:24.596607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.590139 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:39:24.596900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.596878 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 01 22:39:24.597082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.597056 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 01 22:39:24.729597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.728352 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:24.741005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.740950 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:24.744776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.744742 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.756938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.756912 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr"] Nov 01 22:39:24.793922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.793864 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:24.794222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.794203 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:24.824115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.824077 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:24.840867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.840802 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:39:24.842193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.842176 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:39:24.842841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.842826 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:39:24.843106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.843094 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:39:24.946732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:24.938155 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:25.462833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.461315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:25.490158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.490127 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:25.505509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.500791 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" event={"ID":"aad69459-832e-4a63-b7a3-3fb5cd482c4b","Type":"ContainerStarted","Data":"699c3eb6293ff9186516589043e8438d02c8e12f841c6ba85460ef738c3f3d47"} Nov 01 22:39:25.505509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.500917 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:39:25.505509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.502433 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:39:25.505509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.504901 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:39:25.506271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.505976 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:39:25.514798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.514757 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:39:25.546997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.546957 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:39:25.577590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.577535 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" event={"ID":"35583d35-83a2-4e3d-9997-6773a898f70f","Type":"ContainerDied","Data":"48a70b02dbfa0df1f1ab1ee2ad1de0f413068344c345b5e35448147549d42ba5"} Nov 01 22:39:25.577772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.577636 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a70b02dbfa0df1f1ab1ee2ad1de0f413068344c345b5e35448147549d42ba5" Nov 01 22:39:25.577910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.577885 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2" Nov 01 22:39:25.627077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.627046 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:25.657594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.629771 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 22:39:25.677311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.677251 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-xks-ca\") pod \"35583d35-83a2-4e3d-9997-6773a898f70f\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " Nov 01 22:39:25.731394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.731305 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35583d35-83a2-4e3d-9997-6773a898f70f-kube-api-access-mhccb\" (UniqueName: \"kubernetes.io/projected/35583d35-83a2-4e3d-9997-6773a898f70f-kube-api-access-mhccb\") pod \"35583d35-83a2-4e3d-9997-6773a898f70f\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " Nov 01 22:39:25.731702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.731679 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:25.731951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.731929 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-openshift-ca\") pod \"35583d35-83a2-4e3d-9997-6773a898f70f\" (UID: \"35583d35-83a2-4e3d-9997-6773a898f70f\") " Nov 01 22:39:25.734794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.734769 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-openshift-ca" pod "35583d35-83a2-4e3d-9997-6773a898f70f" (UID: "35583d35-83a2-4e3d-9997-6773a898f70f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:25.801711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.801679 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-xks-ca" pod "35583d35-83a2-4e3d-9997-6773a898f70f" (UID: "35583d35-83a2-4e3d-9997-6773a898f70f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:25.801920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.801892 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35583d35-83a2-4e3d-9997-6773a898f70f-kube-api-access-mhccb" pod "35583d35-83a2-4e3d-9997-6773a898f70f" (UID: "35583d35-83a2-4e3d-9997-6773a898f70f"). InnerVolumeSpecName "kube-api-access-mhccb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:25.806947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.806924 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:25.812799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.812778 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:39:25.814739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.814665 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:39:25.835833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.835796 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:25.835833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.835837 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhccb\" (UniqueName: \"kubernetes.io/projected/35583d35-83a2-4e3d-9997-6773a898f70f-kube-api-access-mhccb\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:25.836072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.835854 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35583d35-83a2-4e3d-9997-6773a898f70f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:25.913702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.913659 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:25.920132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:25.920101 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:39:26.493643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:26.491563 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:26.669599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:26.667064 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" event={"ID":"aad69459-832e-4a63-b7a3-3fb5cd482c4b","Type":"ContainerStarted","Data":"b3b23bac32827b37ffbf48e858276273b3b7ef2907f3c146ac69121d8d735ca8"} Nov 01 22:39:26.730661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:26.719959 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" podStartSLOduration=7.719939489 podStartE2EDuration="7.719939489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:26.705871003 +0000 UTC m=+932.133748939" watchObservedRunningTime="2026-11-01 22:39:26.719939489 +0000 UTC m=+932.147817422" Nov 01 22:39:27.677057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:27.677023 4981 generic.go:356] "Generic (PLEG): container finished" podID="aad69459-832e-4a63-b7a3-3fb5cd482c4b" containerID="b3b23bac32827b37ffbf48e858276273b3b7ef2907f3c146ac69121d8d735ca8" exitCode=0 Nov 01 22:39:27.677431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:27.677115 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" event={"ID":"aad69459-832e-4a63-b7a3-3fb5cd482c4b","Type":"ContainerDied","Data":"b3b23bac32827b37ffbf48e858276273b3b7ef2907f3c146ac69121d8d735ca8"} Nov 01 22:39:27.741281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:27.741254 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:28.703337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.703308 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:28.895545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.895520 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:28.968822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.968681 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-openshift-ca\") pod \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " Nov 01 22:39:28.969187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.969153 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-openshift-ca" pod "aad69459-832e-4a63-b7a3-3fb5cd482c4b" (UID: "aad69459-832e-4a63-b7a3-3fb5cd482c4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:28.969460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.969438 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad69459-832e-4a63-b7a3-3fb5cd482c4b-kube-api-access-mhmbr\" (UniqueName: \"kubernetes.io/projected/aad69459-832e-4a63-b7a3-3fb5cd482c4b-kube-api-access-mhmbr\") pod \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " Nov 01 22:39:28.969717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.969661 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-xks-ca\") pod \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\" (UID: \"aad69459-832e-4a63-b7a3-3fb5cd482c4b\") " Nov 01 22:39:28.970472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.970439 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:28.972157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.972130 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad69459-832e-4a63-b7a3-3fb5cd482c4b-kube-api-access-mhmbr" pod "aad69459-832e-4a63-b7a3-3fb5cd482c4b" (UID: "aad69459-832e-4a63-b7a3-3fb5cd482c4b"). InnerVolumeSpecName "kube-api-access-mhmbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:28.973184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:28.973157 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-xks-ca" pod "aad69459-832e-4a63-b7a3-3fb5cd482c4b" (UID: "aad69459-832e-4a63-b7a3-3fb5cd482c4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:29.072220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:29.072184 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmbr\" (UniqueName: \"kubernetes.io/projected/aad69459-832e-4a63-b7a3-3fb5cd482c4b-kube-api-access-mhmbr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:29.072220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:29.072213 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad69459-832e-4a63-b7a3-3fb5cd482c4b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:29.668686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:29.668652 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:29.696386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:29.696347 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" Nov 01 22:39:29.696528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:29.696406 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr" event={"ID":"aad69459-832e-4a63-b7a3-3fb5cd482c4b","Type":"ContainerDied","Data":"699c3eb6293ff9186516589043e8438d02c8e12f841c6ba85460ef738c3f3d47"} Nov 01 22:39:29.696528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:29.696449 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699c3eb6293ff9186516589043e8438d02c8e12f841c6ba85460ef738c3f3d47" Nov 01 22:39:30.080716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.080677 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws"] Nov 01 22:39:30.081646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.081570 4981 cpu_manager.go:414] "containerMap: removing container" podUID="35583d35-83a2-4e3d-9997-6773a898f70f" containerName="revoke-keys" Nov 01 22:39:30.081646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.081610 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35583d35-83a2-4e3d-9997-6773a898f70f" containerName="revoke-keys" Nov 01 22:39:30.081819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.081684 4981 cpu_manager.go:414] "containerMap: removing container" podUID="aad69459-832e-4a63-b7a3-3fb5cd482c4b" containerName="revoke-keys" Nov 01 22:39:30.081819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.081693 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad69459-832e-4a63-b7a3-3fb5cd482c4b" containerName="revoke-keys" Nov 01 22:39:30.081928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.081884 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad69459-832e-4a63-b7a3-3fb5cd482c4b" containerName="revoke-keys" Nov 01 22:39:30.081928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.081907 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="35583d35-83a2-4e3d-9997-6773a898f70f" containerName="revoke-keys" Nov 01 22:39:30.086002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.085978 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.089938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.089915 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:39:30.090070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.089957 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:39:30.091293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.091271 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:39:30.091293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.091286 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:39:30.092913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.092891 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws"] Nov 01 22:39:30.189373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.189333 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.189550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.189460 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsw6g\" (UniqueName: \"kubernetes.io/projected/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-kube-api-access-dsw6g\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.189629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.189542 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.290725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.290687 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsw6g\" (UniqueName: \"kubernetes.io/projected/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-kube-api-access-dsw6g\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.290906 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.290767 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.291065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.291047 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.291412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.291391 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.293140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.293123 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.303802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.303776 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsw6g\" (UniqueName: \"kubernetes.io/projected/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-kube-api-access-dsw6g\") pod \"maas-api-revoke-sub-74a5e869-bq7ws\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.403769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.403684 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:30.473038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.473007 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:30.702712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.702678 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws"] Nov 01 22:39:30.707304 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:30.707273 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9266fac5_94fb_4ba1_82f2_fe7131f1b0de.slice/crio-46f86182eee326448fd37bbd7515484bb751cb250e46b4f88b83a7571ccd6c64 WatchSource:0}: Error finding container 46f86182eee326448fd37bbd7515484bb751cb250e46b4f88b83a7571ccd6c64: Status 404 returned error can't find the container with id 46f86182eee326448fd37bbd7515484bb751cb250e46b4f88b83a7571ccd6c64 Nov 01 22:39:30.863998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.863968 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5"] Nov 01 22:39:30.870113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.870087 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:30.877476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:30.877448 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5"] Nov 01 22:39:31.002928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.002893 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzczx\" (UniqueName: \"kubernetes.io/projected/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-kube-api-access-bzczx\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.003083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.002972 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.003204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.003187 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.104593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.104546 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzczx\" (UniqueName: \"kubernetes.io/projected/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-kube-api-access-bzczx\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.104944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.104652 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.104944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.104911 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.105479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.105455 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.106915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.106893 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.121223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.121201 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzczx\" (UniqueName: \"kubernetes.io/projected/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-kube-api-access-bzczx\") pod \"maas-api-revoke-sub-2346a4fe-cggn5\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.187496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.187467 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:31.374347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.374315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:31.481724 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:31.481690 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d8b2dd_b1b9_4b3a_abc4_7538e938bbf6.slice/crio-b81fe5f93e052f90f79d3b3943a0e16993dc32106d5745a5eb2e78b468a5fb0c WatchSource:0}: Error finding container b81fe5f93e052f90f79d3b3943a0e16993dc32106d5745a5eb2e78b468a5fb0c: Status 404 returned error can't find the container with id b81fe5f93e052f90f79d3b3943a0e16993dc32106d5745a5eb2e78b468a5fb0c Nov 01 22:39:31.484216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.484183 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5"] Nov 01 22:39:31.719816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.719782 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" event={"ID":"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6","Type":"ContainerStarted","Data":"b81fe5f93e052f90f79d3b3943a0e16993dc32106d5745a5eb2e78b468a5fb0c"} Nov 01 22:39:31.721237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.721203 4981 generic.go:356] "Generic (PLEG): container finished" podID="9266fac5-94fb-4ba1-82f2-fe7131f1b0de" containerID="99e82a00787c9d41542ce95af1b6437b2fa5f6eca412af2e2eab79121dd5a833" exitCode=0 Nov 01 22:39:31.721367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.721256 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" event={"ID":"9266fac5-94fb-4ba1-82f2-fe7131f1b0de","Type":"ContainerDied","Data":"99e82a00787c9d41542ce95af1b6437b2fa5f6eca412af2e2eab79121dd5a833"} Nov 01 22:39:31.721367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:31.721301 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" event={"ID":"9266fac5-94fb-4ba1-82f2-fe7131f1b0de","Type":"ContainerStarted","Data":"46f86182eee326448fd37bbd7515484bb751cb250e46b4f88b83a7571ccd6c64"} Nov 01 22:39:32.285454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:32.285422 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:32.732021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:32.731987 4981 generic.go:356] "Generic (PLEG): container finished" podID="93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" containerID="55cdf8b2b78b029002969f1c44c0fc0a6c56c73f9550ce19fd014650983e0b4a" exitCode=0 Nov 01 22:39:32.735091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:32.735064 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" event={"ID":"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6","Type":"ContainerDied","Data":"55cdf8b2b78b029002969f1c44c0fc0a6c56c73f9550ce19fd014650983e0b4a"} Nov 01 22:39:32.897110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:32.897086 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:33.034199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.034097 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-openshift-ca\") pod \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " Nov 01 22:39:33.034348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.034320 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-xks-ca\") pod \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " Nov 01 22:39:33.034400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.034348 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-kube-api-access-dsw6g\" (UniqueName: \"kubernetes.io/projected/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-kube-api-access-dsw6g\") pod \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\" (UID: \"9266fac5-94fb-4ba1-82f2-fe7131f1b0de\") " Nov 01 22:39:33.034469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.034445 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-openshift-ca" pod "9266fac5-94fb-4ba1-82f2-fe7131f1b0de" (UID: "9266fac5-94fb-4ba1-82f2-fe7131f1b0de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:33.035042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.035006 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:33.036720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.036696 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-xks-ca" pod "9266fac5-94fb-4ba1-82f2-fe7131f1b0de" (UID: "9266fac5-94fb-4ba1-82f2-fe7131f1b0de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:33.036823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.036744 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-kube-api-access-dsw6g" pod "9266fac5-94fb-4ba1-82f2-fe7131f1b0de" (UID: "9266fac5-94fb-4ba1-82f2-fe7131f1b0de"). InnerVolumeSpecName "kube-api-access-dsw6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:33.054145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.054119 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx"] Nov 01 22:39:33.054843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.054828 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9266fac5-94fb-4ba1-82f2-fe7131f1b0de" containerName="revoke-keys" Nov 01 22:39:33.054897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.054844 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9266fac5-94fb-4ba1-82f2-fe7131f1b0de" containerName="revoke-keys" Nov 01 22:39:33.054996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.054987 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9266fac5-94fb-4ba1-82f2-fe7131f1b0de" containerName="revoke-keys" Nov 01 22:39:33.058860 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.058840 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.068420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.068388 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx"] Nov 01 22:39:33.085276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.085251 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:33.155324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.155254 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.155324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.155324 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xhk\" (UniqueName: \"kubernetes.io/projected/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-kube-api-access-25xhk\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.155704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.155433 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.155774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.155744 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:33.155774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.155766 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsw6g\" (UniqueName: \"kubernetes.io/projected/9266fac5-94fb-4ba1-82f2-fe7131f1b0de-kube-api-access-dsw6g\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:33.256685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.256649 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.256881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.256693 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xhk\" (UniqueName: \"kubernetes.io/projected/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-kube-api-access-25xhk\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.256881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.256739 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.257413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.257385 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.259205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.259182 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.264248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.264229 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xhk\" (UniqueName: \"kubernetes.io/projected/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-kube-api-access-25xhk\") pod \"maas-api-revoke-sub-bca42470-rw4kx\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.380587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.380507 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:33.515818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.515776 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n"] Nov 01 22:39:33.531558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.529805 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.545668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.545638 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n"] Nov 01 22:39:33.663275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.663247 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.663366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.663354 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.663499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.663485 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfxb\" (UniqueName: \"kubernetes.io/projected/b02e814b-1913-428b-b40e-f8c5ecee320b-kube-api-access-rxfxb\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.666645 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:33.666603 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2dfc48_2818_4dc9_a3e2_0bac45112c37.slice/crio-ec8e5d07ace189a6a944d7fa57ffa232f45b873f9a0b0e1721d36a2837199126 WatchSource:0}: Error finding container ec8e5d07ace189a6a944d7fa57ffa232f45b873f9a0b0e1721d36a2837199126: Status 404 returned error can't find the container with id ec8e5d07ace189a6a944d7fa57ffa232f45b873f9a0b0e1721d36a2837199126 Nov 01 22:39:33.669524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.669502 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx"] Nov 01 22:39:33.742351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.742321 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" event={"ID":"9266fac5-94fb-4ba1-82f2-fe7131f1b0de","Type":"ContainerDied","Data":"46f86182eee326448fd37bbd7515484bb751cb250e46b4f88b83a7571ccd6c64"} Nov 01 22:39:33.742462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.742354 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws" Nov 01 22:39:33.742462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.742363 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46f86182eee326448fd37bbd7515484bb751cb250e46b4f88b83a7571ccd6c64" Nov 01 22:39:33.743810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.743788 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" event={"ID":"5b2dfc48-2818-4dc9-a3e2-0bac45112c37","Type":"ContainerStarted","Data":"ec8e5d07ace189a6a944d7fa57ffa232f45b873f9a0b0e1721d36a2837199126"} Nov 01 22:39:33.764891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.764865 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.765036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.764933 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.765036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.765027 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfxb\" (UniqueName: \"kubernetes.io/projected/b02e814b-1913-428b-b40e-f8c5ecee320b-kube-api-access-rxfxb\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.765591 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.765550 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.767439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.767414 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.772995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.772972 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfxb\" (UniqueName: \"kubernetes.io/projected/b02e814b-1913-428b-b40e-f8c5ecee320b-kube-api-access-rxfxb\") pod \"maas-api-revoke-sub-7b34ab10-6gx7n\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.855612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.855571 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:33.917584 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.917517 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:33.994211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.994177 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7"] Nov 01 22:39:33.997719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.997674 4981 cpu_manager.go:414] "containerMap: removing container" podUID="93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" containerName="revoke-keys" Nov 01 22:39:33.997719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.997720 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" containerName="revoke-keys" Nov 01 22:39:33.998052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:33.998035 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" containerName="revoke-keys" Nov 01 22:39:34.006752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.006731 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7"] Nov 01 22:39:34.006918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.006894 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.041829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.041792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:34.069984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.069947 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-kube-api-access-bzczx\" (UniqueName: \"kubernetes.io/projected/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-kube-api-access-bzczx\") pod \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " Nov 01 22:39:34.070249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.070230 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-xks-ca\") pod \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " Nov 01 22:39:34.070366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.070352 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-openshift-ca\") pod \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\" (UID: \"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6\") " Nov 01 22:39:34.070739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.070713 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-openshift-ca" pod "93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" (UID: "93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:34.070954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.070935 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:34.073950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.073928 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-xks-ca" pod "93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" (UID: "93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:34.074039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.073960 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-kube-api-access-bzczx" pod "93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" (UID: "93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6"). InnerVolumeSpecName "kube-api-access-bzczx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:34.152400 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:34.152362 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02e814b_1913_428b_b40e_f8c5ecee320b.slice/crio-f45615bcb51be8999c2ceeebafaabec15ad545d8fe22a92b2f00ce8df3dcf411 WatchSource:0}: Error finding container f45615bcb51be8999c2ceeebafaabec15ad545d8fe22a92b2f00ce8df3dcf411: Status 404 returned error can't find the container with id f45615bcb51be8999c2ceeebafaabec15ad545d8fe22a92b2f00ce8df3dcf411 Nov 01 22:39:34.157984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.156380 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n"] Nov 01 22:39:34.174207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.173498 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.174207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.173625 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.174207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.173818 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mstj6\" (UniqueName: \"kubernetes.io/projected/c33037fa-36a4-4475-bc97-a573ac90893d-kube-api-access-mstj6\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.174207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.173980 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:34.174207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.173997 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzczx\" (UniqueName: \"kubernetes.io/projected/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6-kube-api-access-bzczx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:34.275157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.275121 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mstj6\" (UniqueName: \"kubernetes.io/projected/c33037fa-36a4-4475-bc97-a573ac90893d-kube-api-access-mstj6\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.275321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.275231 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.275407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.275318 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.276053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.276022 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.277721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.277698 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.284497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.284477 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mstj6\" (UniqueName: \"kubernetes.io/projected/c33037fa-36a4-4475-bc97-a573ac90893d-kube-api-access-mstj6\") pod \"maas-api-revoke-sub-c9c8e53b-f2pz7\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.330994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.330968 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:34.618240 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:34.618209 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33037fa_36a4_4475_bc97_a573ac90893d.slice/crio-f13e5a80c25f11bf70543981b8b769d80878881f221a237cd8720afb2df75876 WatchSource:0}: Error finding container f13e5a80c25f11bf70543981b8b769d80878881f221a237cd8720afb2df75876: Status 404 returned error can't find the container with id f13e5a80c25f11bf70543981b8b769d80878881f221a237cd8720afb2df75876 Nov 01 22:39:34.619718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.619689 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7"] Nov 01 22:39:34.752735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.752710 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" Nov 01 22:39:34.752735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.752721 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5" event={"ID":"93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6","Type":"ContainerDied","Data":"b81fe5f93e052f90f79d3b3943a0e16993dc32106d5745a5eb2e78b468a5fb0c"} Nov 01 22:39:34.752890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.752772 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81fe5f93e052f90f79d3b3943a0e16993dc32106d5745a5eb2e78b468a5fb0c" Nov 01 22:39:34.754522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.754499 4981 generic.go:356] "Generic (PLEG): container finished" podID="5b2dfc48-2818-4dc9-a3e2-0bac45112c37" containerID="0960ce90f44e9ce5e6a547129385c3e2a6453cfdcd1924e8f701cf82ed7db85b" exitCode=0 Nov 01 22:39:34.754642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.754600 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" event={"ID":"5b2dfc48-2818-4dc9-a3e2-0bac45112c37","Type":"ContainerDied","Data":"0960ce90f44e9ce5e6a547129385c3e2a6453cfdcd1924e8f701cf82ed7db85b"} Nov 01 22:39:34.756707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.756684 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" event={"ID":"b02e814b-1913-428b-b40e-f8c5ecee320b","Type":"ContainerStarted","Data":"6e3e52b610d2f5035055fe168acecf35e50fd75746c7569addf6f13445ba32af"} Nov 01 22:39:34.756808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.756725 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" event={"ID":"b02e814b-1913-428b-b40e-f8c5ecee320b","Type":"ContainerStarted","Data":"f45615bcb51be8999c2ceeebafaabec15ad545d8fe22a92b2f00ce8df3dcf411"} Nov 01 22:39:34.758116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.758097 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" event={"ID":"c33037fa-36a4-4475-bc97-a573ac90893d","Type":"ContainerStarted","Data":"f13e5a80c25f11bf70543981b8b769d80878881f221a237cd8720afb2df75876"} Nov 01 22:39:34.798263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.798238 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:34.815321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:34.815270 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" podStartSLOduration=1.8152538150000002 podStartE2EDuration="1.815253815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:34.807333963 +0000 UTC m=+940.235211889" watchObservedRunningTime="2026-11-01 22:39:34.815253815 +0000 UTC m=+940.243131728" Nov 01 22:39:35.560054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:35.560023 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:35.768741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:35.768705 4981 generic.go:356] "Generic (PLEG): container finished" podID="b02e814b-1913-428b-b40e-f8c5ecee320b" containerID="6e3e52b610d2f5035055fe168acecf35e50fd75746c7569addf6f13445ba32af" exitCode=0 Nov 01 22:39:35.769166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:35.768751 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" event={"ID":"b02e814b-1913-428b-b40e-f8c5ecee320b","Type":"ContainerDied","Data":"6e3e52b610d2f5035055fe168acecf35e50fd75746c7569addf6f13445ba32af"} Nov 01 22:39:35.770445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:35.770421 4981 generic.go:356] "Generic (PLEG): container finished" podID="c33037fa-36a4-4475-bc97-a573ac90893d" containerID="5507dc7fb63b7b7e23bc13c2b6cd9b39f833abcbc02f9813a5a9af6d1aa0bb5d" exitCode=0 Nov 01 22:39:35.770601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:35.770515 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" event={"ID":"c33037fa-36a4-4475-bc97-a573ac90893d","Type":"ContainerDied","Data":"5507dc7fb63b7b7e23bc13c2b6cd9b39f833abcbc02f9813a5a9af6d1aa0bb5d"} Nov 01 22:39:35.944251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:35.944226 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:36.010118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.010077 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-xks-ca\") pod \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " Nov 01 22:39:36.010317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.010178 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-kube-api-access-25xhk\" (UniqueName: \"kubernetes.io/projected/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-kube-api-access-25xhk\") pod \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " Nov 01 22:39:36.011498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.010400 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-openshift-ca\") pod \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\" (UID: \"5b2dfc48-2818-4dc9-a3e2-0bac45112c37\") " Nov 01 22:39:36.011498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.011421 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-openshift-ca" pod "5b2dfc48-2818-4dc9-a3e2-0bac45112c37" (UID: "5b2dfc48-2818-4dc9-a3e2-0bac45112c37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:36.012701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.012672 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-kube-api-access-25xhk" pod "5b2dfc48-2818-4dc9-a3e2-0bac45112c37" (UID: "5b2dfc48-2818-4dc9-a3e2-0bac45112c37"). InnerVolumeSpecName "kube-api-access-25xhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:36.012909 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.012880 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-xks-ca" pod "5b2dfc48-2818-4dc9-a3e2-0bac45112c37" (UID: "5b2dfc48-2818-4dc9-a3e2-0bac45112c37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:36.112457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.112374 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:36.112457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.112406 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25xhk\" (UniqueName: \"kubernetes.io/projected/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-kube-api-access-25xhk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:36.112457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.112417 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b2dfc48-2818-4dc9-a3e2-0bac45112c37-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:36.279000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.278971 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:36.779677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.779653 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" Nov 01 22:39:36.780038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.779691 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx" event={"ID":"5b2dfc48-2818-4dc9-a3e2-0bac45112c37","Type":"ContainerDied","Data":"ec8e5d07ace189a6a944d7fa57ffa232f45b873f9a0b0e1721d36a2837199126"} Nov 01 22:39:36.780038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.779734 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec8e5d07ace189a6a944d7fa57ffa232f45b873f9a0b0e1721d36a2837199126" Nov 01 22:39:36.956632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.956603 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:36.968192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:36.968164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:37.032644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.028180 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33037fa-36a4-4475-bc97-a573ac90893d-kube-api-access-mstj6\" (UniqueName: \"kubernetes.io/projected/c33037fa-36a4-4475-bc97-a573ac90893d-kube-api-access-mstj6\") pod \"c33037fa-36a4-4475-bc97-a573ac90893d\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " Nov 01 22:39:37.032644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.028431 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-xks-ca\") pod \"c33037fa-36a4-4475-bc97-a573ac90893d\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " Nov 01 22:39:37.032644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.028524 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-openshift-ca\") pod \"c33037fa-36a4-4475-bc97-a573ac90893d\" (UID: \"c33037fa-36a4-4475-bc97-a573ac90893d\") " Nov 01 22:39:37.032644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.030019 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-openshift-ca" pod "c33037fa-36a4-4475-bc97-a573ac90893d" (UID: "c33037fa-36a4-4475-bc97-a573ac90893d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:37.050775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.040112 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33037fa-36a4-4475-bc97-a573ac90893d-kube-api-access-mstj6" pod "c33037fa-36a4-4475-bc97-a573ac90893d" (UID: "c33037fa-36a4-4475-bc97-a573ac90893d"). InnerVolumeSpecName "kube-api-access-mstj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:37.050775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.040656 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-xks-ca" pod "c33037fa-36a4-4475-bc97-a573ac90893d" (UID: "c33037fa-36a4-4475-bc97-a573ac90893d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:37.050775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.042751 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:37.130448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.130418 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-openshift-ca\") pod \"b02e814b-1913-428b-b40e-f8c5ecee320b\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " Nov 01 22:39:37.130607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.130476 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-xks-ca\") pod \"b02e814b-1913-428b-b40e-f8c5ecee320b\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " Nov 01 22:39:37.130654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.130637 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02e814b-1913-428b-b40e-f8c5ecee320b-kube-api-access-rxfxb\" (UniqueName: \"kubernetes.io/projected/b02e814b-1913-428b-b40e-f8c5ecee320b-kube-api-access-rxfxb\") pod \"b02e814b-1913-428b-b40e-f8c5ecee320b\" (UID: \"b02e814b-1913-428b-b40e-f8c5ecee320b\") " Nov 01 22:39:37.130863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.130840 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-openshift-ca" pod "b02e814b-1913-428b-b40e-f8c5ecee320b" (UID: "b02e814b-1913-428b-b40e-f8c5ecee320b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:37.131218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.131197 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mstj6\" (UniqueName: \"kubernetes.io/projected/c33037fa-36a4-4475-bc97-a573ac90893d-kube-api-access-mstj6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:37.131218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.131218 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:37.131355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.131228 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:37.131355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.131237 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c33037fa-36a4-4475-bc97-a573ac90893d-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:37.132701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.132674 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-xks-ca" pod "b02e814b-1913-428b-b40e-f8c5ecee320b" (UID: "b02e814b-1913-428b-b40e-f8c5ecee320b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:37.132792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.132735 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02e814b-1913-428b-b40e-f8c5ecee320b-kube-api-access-rxfxb" pod "b02e814b-1913-428b-b40e-f8c5ecee320b" (UID: "b02e814b-1913-428b-b40e-f8c5ecee320b"). InnerVolumeSpecName "kube-api-access-rxfxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:37.232383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.232354 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfxb\" (UniqueName: \"kubernetes.io/projected/b02e814b-1913-428b-b40e-f8c5ecee320b-kube-api-access-rxfxb\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:37.232383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.232378 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b02e814b-1913-428b-b40e-f8c5ecee320b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:37.709606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.709550 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:37.797884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.797430 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" event={"ID":"b02e814b-1913-428b-b40e-f8c5ecee320b","Type":"ContainerDied","Data":"f45615bcb51be8999c2ceeebafaabec15ad545d8fe22a92b2f00ce8df3dcf411"} Nov 01 22:39:37.798524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.798346 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n" Nov 01 22:39:37.798524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.798512 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f45615bcb51be8999c2ceeebafaabec15ad545d8fe22a92b2f00ce8df3dcf411" Nov 01 22:39:37.800911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.800889 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" Nov 01 22:39:37.801108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.800921 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7" event={"ID":"c33037fa-36a4-4475-bc97-a573ac90893d","Type":"ContainerDied","Data":"f13e5a80c25f11bf70543981b8b769d80878881f221a237cd8720afb2df75876"} Nov 01 22:39:37.801199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:37.801132 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f13e5a80c25f11bf70543981b8b769d80878881f221a237cd8720afb2df75876" Nov 01 22:39:38.275886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.275854 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l"] Nov 01 22:39:38.276717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276692 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5b2dfc48-2818-4dc9-a3e2-0bac45112c37" containerName="revoke-keys" Nov 01 22:39:38.276717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276713 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b2dfc48-2818-4dc9-a3e2-0bac45112c37" containerName="revoke-keys" Nov 01 22:39:38.276962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276736 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b02e814b-1913-428b-b40e-f8c5ecee320b" containerName="revoke-keys" Nov 01 22:39:38.276962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276742 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02e814b-1913-428b-b40e-f8c5ecee320b" containerName="revoke-keys" Nov 01 22:39:38.276962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276755 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c33037fa-36a4-4475-bc97-a573ac90893d" containerName="revoke-keys" Nov 01 22:39:38.276962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276760 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33037fa-36a4-4475-bc97-a573ac90893d" containerName="revoke-keys" Nov 01 22:39:38.276962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276889 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b2dfc48-2818-4dc9-a3e2-0bac45112c37" containerName="revoke-keys" Nov 01 22:39:38.276962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276902 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02e814b-1913-428b-b40e-f8c5ecee320b" containerName="revoke-keys" Nov 01 22:39:38.276962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.276910 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33037fa-36a4-4475-bc97-a573ac90893d" containerName="revoke-keys" Nov 01 22:39:38.280956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.280937 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.306181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.306015 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l"] Nov 01 22:39:38.348868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.348835 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.348978 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.349106 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.349137 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.349222 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74gm\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-kube-api-access-v74gm\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.349272 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.349305 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.349459 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.349534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.349500 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.388783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.388742 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk"] Nov 01 22:39:38.425523 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.425487 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk"] Nov 01 22:39:38.425816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.425636 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.451268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.451243 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.451461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.451445 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74gm\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-kube-api-access-v74gm\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.451621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.451564 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.451764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.451744 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.451970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.451908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7vg\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-kube-api-access-8w7vg\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.451977 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.452080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452006 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.452080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452050 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452155 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452196 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.452362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452253 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452277 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5987c39-5892-4330-abf3-1694e5668153-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452329 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452360 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452435 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.452564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452518 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5987c39-5892-4330-abf3-1694e5668153-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452555 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.452824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452689 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-data\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.452893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.452875 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-credential-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.453115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.453093 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-socket\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.453344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.453324 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.453512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.453490 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-certs\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.453658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.453633 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istiod-ca-cert\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.455668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.455641 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-envoy\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.456187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.456166 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-podinfo\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.460978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.460958 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74gm\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-kube-api-access-v74gm\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.462311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.462290 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-token\") pod \"e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.495099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.495075 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:38.555220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555111 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7vg\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-kube-api-access-8w7vg\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555212 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555296 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555358 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555383 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5987c39-5892-4330-abf3-1694e5668153-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555430 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555458 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555592 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5987c39-5892-4330-abf3-1694e5668153-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.555763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.555619 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.558373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.556816 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-certs\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.558373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.557214 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-data\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.558757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.558728 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-credential-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.559301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.559276 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-socket\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.559930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.559901 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5987c39-5892-4330-abf3-1694e5668153-istiod-ca-cert\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.563096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.563072 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5987c39-5892-4330-abf3-1694e5668153-istio-podinfo\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.563208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.563044 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-envoy\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.565650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.565629 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-istio-token\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.566357 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.566339 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7vg\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-kube-api-access-8w7vg\") pod \"e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.599279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.599246 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:38.762283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.762254 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:38.902387 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:38.902353 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e4d4eb_eb65_47d5_ae4e_348b29afbe85.slice/crio-d48f87e971f1df28883b655c54cf334371069fdc04a5adcb5a97c489e3e8c142 WatchSource:0}: Error finding container d48f87e971f1df28883b655c54cf334371069fdc04a5adcb5a97c489e3e8c142: Status 404 returned error can't find the container with id d48f87e971f1df28883b655c54cf334371069fdc04a5adcb5a97c489e3e8c142 Nov 01 22:39:38.905971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.905939 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l"] Nov 01 22:39:38.906062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.905935 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:38.906135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.906106 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:38.906185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:38.906164 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.056739 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:39.056697 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5987c39_5892_4330_abf3_1694e5668153.slice/crio-be01546f73d9ef5f93b12dd1a4bd8575f41266483222c6f1479e1963a1400feb WatchSource:0}: Error finding container be01546f73d9ef5f93b12dd1a4bd8575f41266483222c6f1479e1963a1400feb: Status 404 returned error can't find the container with id be01546f73d9ef5f93b12dd1a4bd8575f41266483222c6f1479e1963a1400feb Nov 01 22:39:39.059613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.059562 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.059722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.059650 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.059722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.059683 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.060743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.060716 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk"] Nov 01 22:39:39.132045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.132016 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq"] Nov 01 22:39:39.146034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.139434 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.164708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.164673 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq"] Nov 01 22:39:39.166343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166313 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.166483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166464 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.166551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166527 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.166623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166592 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.166681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166667 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.166802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166783 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhhq\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-kube-api-access-5vhhq\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.166884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166854 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.166948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.167002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.166986 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.182507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.182472 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4"] Nov 01 22:39:39.187711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.187689 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4"] Nov 01 22:39:39.187832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.187814 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.268487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268436 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.268687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268494 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.268687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268544 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.268687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268607 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.268687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268679 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.268929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268754 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhhq\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-kube-api-access-5vhhq\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.268929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268812 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.268929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268858 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.268929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268890 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268934 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.268989 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269014 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269042 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzkm\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-kube-api-access-rvzkm\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269091 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269157 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269211 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269244 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269310 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.269855 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.270486 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.270890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.270796 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.271819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.271796 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.272186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.272167 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.273842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.273818 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.274901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.274870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.281207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.281182 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.282526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.282504 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhhq\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-kube-api-access-5vhhq\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.373619 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.373721 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.373801 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.373834 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzkm\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-kube-api-access-rvzkm\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.373894 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.373996 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.374039 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.374118 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.374210 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.377002 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-certs\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.377495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.377311 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-data\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.379190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.378721 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istiod-ca-cert\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.379190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.378996 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-credential-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.379190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.379151 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-socket\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.379835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.379806 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-envoy\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.379930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.379911 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-podinfo\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.387307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.387283 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-token\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.392054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.392029 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzkm\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-kube-api-access-rvzkm\") pod \"e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.467070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.467036 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:39.503454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.503423 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:39.541745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.541705 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:39.830472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.830411 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" event={"ID":"21e4d4eb-eb65-47d5-ae4e-348b29afbe85","Type":"ContainerStarted","Data":"2fa7b5d3c9f255115424b50bd23e77a7f6e97e4556499d69e0cee9a71935e10a"} Nov 01 22:39:39.830713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.830514 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" event={"ID":"21e4d4eb-eb65-47d5-ae4e-348b29afbe85","Type":"ContainerStarted","Data":"d48f87e971f1df28883b655c54cf334371069fdc04a5adcb5a97c489e3e8c142"} Nov 01 22:39:39.835482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.835456 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq"] Nov 01 22:39:39.838604 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:39.838554 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc735e8a_9d22_4cbc_868b_be4de72ccbe9.slice/crio-3e6dccc6bcf992f67fe6a3a9c4a7e1d655a98cd4ddedbede99de22bf9ec658af WatchSource:0}: Error finding container 3e6dccc6bcf992f67fe6a3a9c4a7e1d655a98cd4ddedbede99de22bf9ec658af: Status 404 returned error can't find the container with id 3e6dccc6bcf992f67fe6a3a9c4a7e1d655a98cd4ddedbede99de22bf9ec658af Nov 01 22:39:39.839532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.839504 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.839623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.839589 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.839680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.839631 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.841274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.840600 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" event={"ID":"c5987c39-5892-4330-abf3-1694e5668153","Type":"ContainerStarted","Data":"560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334"} Nov 01 22:39:39.841274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.840654 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" event={"ID":"c5987c39-5892-4330-abf3-1694e5668153","Type":"ContainerStarted","Data":"be01546f73d9ef5f93b12dd1a4bd8575f41266483222c6f1479e1963a1400feb"} Nov 01 22:39:39.860262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.860216 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podStartSLOduration=1.860203556 podStartE2EDuration="1.860203556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:39.854696882 +0000 UTC m=+945.282574813" watchObservedRunningTime="2026-11-01 22:39:39.860203556 +0000 UTC m=+945.288081488" Nov 01 22:39:39.905946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.885439 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podStartSLOduration=1.885421486 podStartE2EDuration="1.885421486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:39.884032514 +0000 UTC m=+945.311910446" watchObservedRunningTime="2026-11-01 22:39:39.885421486 +0000 UTC m=+945.313299420" Nov 01 22:39:39.909497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.909467 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4"] Nov 01 22:39:39.913832 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:39.913799 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d11122e_79a5_4d13_a813_3d4c64ab1a9c.slice/crio-f031d1b7edca05bcb0813890e6d8bd587289da287b57491e5c4ec7b583337b18 WatchSource:0}: Error finding container f031d1b7edca05bcb0813890e6d8bd587289da287b57491e5c4ec7b583337b18: Status 404 returned error can't find the container with id f031d1b7edca05bcb0813890e6d8bd587289da287b57491e5c4ec7b583337b18 Nov 01 22:39:39.917683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.917653 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.917759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.917719 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:39.917759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:39.917749 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:39:40.605601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.602327 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:40.605601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.602526 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:40.605601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.602594 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:40.700035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.700000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:40.762715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.762677 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:40.768024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.767996 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:40.768155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.768037 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:40.964246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.964206 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" event={"ID":"5d11122e-79a5-4d13-a813-3d4c64ab1a9c","Type":"ContainerStarted","Data":"2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e"} Nov 01 22:39:40.964920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.964893 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" event={"ID":"5d11122e-79a5-4d13-a813-3d4c64ab1a9c","Type":"ContainerStarted","Data":"f031d1b7edca05bcb0813890e6d8bd587289da287b57491e5c4ec7b583337b18"} Nov 01 22:39:40.968548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.968521 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" event={"ID":"dc735e8a-9d22-4cbc-868b-be4de72ccbe9","Type":"ContainerStarted","Data":"b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21"} Nov 01 22:39:40.968923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:40.968901 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" event={"ID":"dc735e8a-9d22-4cbc-868b-be4de72ccbe9","Type":"ContainerStarted","Data":"3e6dccc6bcf992f67fe6a3a9c4a7e1d655a98cd4ddedbede99de22bf9ec658af"} Nov 01 22:39:41.065473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.043640 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podStartSLOduration=2.04361758 podStartE2EDuration="2.04361758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:40.992761651 +0000 UTC m=+946.420639583" watchObservedRunningTime="2026-11-01 22:39:41.04361758 +0000 UTC m=+946.471495511" Nov 01 22:39:41.103991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.103923 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podStartSLOduration=2.103901644 podStartE2EDuration="2.103901644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:41.066765469 +0000 UTC m=+946.494643382" watchObservedRunningTime="2026-11-01 22:39:41.103901644 +0000 UTC m=+946.531779577" Nov 01 22:39:41.519995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.519955 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:41.520474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.520450 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.520663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.520636 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:41.548344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.548314 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:41.550224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.550197 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.550444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.550418 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:41.616715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.615079 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.616715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.615137 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:41.764179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.764144 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:41.764439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:41.764409 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:42.104602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.098200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:42.173635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.173549 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:39:42.422660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.406521 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:42.432768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.432714 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.434154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.434107 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.511443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.510997 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.511443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.511062 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:42.527428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.526692 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:42.545091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.545056 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.545381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.545351 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:42.602099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.602063 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.602394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.602350 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:42.616717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.616665 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.650260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.650216 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.665279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.664651 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.701663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.700663 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.767601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.767101 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:42.767601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.767176 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:42.840598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.832667 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:42.840598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.833871 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:42.988661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:42.987788 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.020599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.017719 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.288657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.277658 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.416647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.413670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.418736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.418710 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:39:43.440831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.440795 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.441540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.441491 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:43.442296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.442278 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:39:43.442463 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.442451 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:39:43.528606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.507619 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.528606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.507674 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:43.548651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.531747 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.548651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.532177 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:39:43.570600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.559724 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.570600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.559781 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:43.766927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.725672 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.766927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.726186 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:43.766927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.727527 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:39:43.766927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.727683 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:39:43.766927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.728978 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.766927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.729011 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:43.775664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.768051 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:43.775664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.768480 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:39:43.775664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.769463 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.805696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.795736 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:39:43.847266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.824023 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:43.847266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.804673 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:43.847266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.825781 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:43.847266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.835619 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:43.847266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.835957 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:39:43.886027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.858296 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:43.886027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.858687 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:39:43.956620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:43.952798 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:44.051193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.002676 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:44.051193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.003066 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:39:44.051193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.047677 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.051193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.048063 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:39:44.427257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.426857 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:44.490874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.490813 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:44.492486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.492442 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.522520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.522461 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.545552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.545511 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.598382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.598345 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.826032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.825932 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:44.838988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.819104 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:44.840615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.767964 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:44.847961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.600616 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.848314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.848260 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:44.848457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.616953 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.848596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.848557 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:44.848722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.817730 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:44.848839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.848820 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:44.848976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.819135 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:44.862948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.858643 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:44.864484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.864437 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:44.882294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.874802 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.882294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.819156 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:44.882294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.819250 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.902183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.892733 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:44.902183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.892811 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Nov 01 22:39:44.954495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:44.954433 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.000897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.000851 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.048762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.048706 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.082043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.081939 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.333997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.196086 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.467108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.467056 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.701588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.701534 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:39:45.703237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.498096 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:45.723003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.546048 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.724541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.576775 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.724815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.724770 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:45.724985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.577195 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.725135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.725110 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:45.725273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.659280 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.725412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.725392 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:45.827782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.827737 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:45.828072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.828039 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:45.828944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.497445 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.855355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.855313 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:45.857289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.855600 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:45.858497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.855631 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:45.884643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.872214 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.885055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.885022 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:39:45.887638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.887603 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.887835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.887666 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:45.887979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.887716 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:46.013720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.920882 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.013988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:45.965978 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.037606 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:39:45.983106 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:39:46.066198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.066157 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.251632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.235262 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.345652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.345538 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.346631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.346595 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:39:46.455611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.455549 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.543011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.537464 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.543011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.537535 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:46.581440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.581386 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.608597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.608483 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.608869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.608822 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:46.705601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.697137 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.765604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.758123 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 01 22:39:46.794875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.774453 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:46.872612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.872497 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:46.872857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.808942 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.873555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.697206 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:46.873794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.847850 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:46.874198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.809200 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:46.874593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.858764 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.874973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.858846 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.875357 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.858879 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:46.901797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.887914 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.901797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.888193 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:46.901797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.888244 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.905265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.905231 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:46.944603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.920856 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:46.948593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:46.946719 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.026845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.026773 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.040247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.040195 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.082904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.082862 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:47.359609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.336145 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:47.570602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.361902 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.571545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.551221 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:47.571777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.571746 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.590601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.586605 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:47.895685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.647172 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.896248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.896209 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:47.906652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.647231 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.906997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.906962 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:47.925078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.648204 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:47.933021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.870088 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.073363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.073322 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:48.073695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975368 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:48.085821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975316 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:48.086487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975389 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:48.086997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975419 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:48.087454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975443 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:48.087919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975470 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:48.088345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975547 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.088523 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975635 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:48.088972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975616 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.089436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.975662 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:48.108111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:47.937482 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:48.108338 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:39:48.071903 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.305s" Nov 01 22:39:48.109997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.072654 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:48.112651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.112621 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.132047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.131996 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.367601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.352983 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.554628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.510965 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.554628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.511124 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:48.577323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.577286 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.578596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.578286 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.582734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.578560 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:48.618669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.612013 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.618669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.612077 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:48.640822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.640669 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:48.640822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.640732 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.768602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.725135 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:48.768602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.766390 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:48.768602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:48.766440 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:49.075713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.075665 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.089751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.089703 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.090519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.090001 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:49.099064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.090030 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:49.099801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.090050 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:49.110225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.090072 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:39:49.114015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.090100 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:49.121951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.090119 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:49.132640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.108668 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.133526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.113640 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:49.138143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.113698 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.138549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.137798 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.334660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.333667 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.412602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.406744 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.504048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.504006 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.504330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.504297 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:49.545012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.544974 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.545303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.545270 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:49.589805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.589683 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.605302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.605268 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.605532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.605508 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:49.740766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.740714 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:49.803396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.803358 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:49.808602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.803670 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:49.991715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.991677 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:49.991966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:49.991936 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.080605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.079742 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.098219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.097657 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:50.102690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.102649 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.104206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.103820 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.112753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.112717 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.116607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.114059 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.142613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.137670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:50.142613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.138077 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.142613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.139782 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:39:50.142613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.140402 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:50.142613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.142179 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.142613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.142411 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.286655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.285764 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:50.286655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.285832 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.286655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.286306 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:50.286655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.286345 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.413601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.412696 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.462602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.458689 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 22:39:50.516602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.516043 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.516602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.516107 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:50.543372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.543261 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.543705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.543624 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:50.592871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.592813 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.600424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.600378 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.600741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.600704 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:50.746478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.746429 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.748021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.746755 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:50.777614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.776780 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:50.777614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:50.776848 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:51.084726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.084675 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.106274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.103319 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.118118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.107567 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.118118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.108356 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.154822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.135640 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.154822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.136607 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.180261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.180217 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.180873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.180838 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.181393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.181369 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:39:51.267651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.224680 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.267651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.226393 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.267651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.228357 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:51.284466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.283406 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:51.440666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.440614 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:51.519854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.519804 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.520058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.519881 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:51.548333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.546749 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.548333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.546809 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:51.597602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.596145 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.616617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.603456 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.616617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.603519 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 01 22:39:51.701657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.685786 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:51.701657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.685882 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:51.760139 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.759493 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:51.795632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.770244 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:51.795632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:51.770302 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:52.110005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.109736 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.111502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.109829 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.111502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.109881 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:52.128281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.128225 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:39:52.136764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.136720 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.138077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.138043 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:52.180704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.180659 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.181438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.181408 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.182327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.182068 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.229606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.229544 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:52.231214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.230853 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.284064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.284017 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:39:52.443245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.443194 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:39:52.507612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.507519 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.507824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.507610 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:52.562609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.561483 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.562609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.561543 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:52.565144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.565119 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:39:52.598998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.598955 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:52.811421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.811328 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:39:52.840613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.824684 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 01 22:39:52.840893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.840853 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 01 22:39:52.937099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.914521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:52.978933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.978889 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:39:52.979768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:52.979750 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:39:53.129686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.129512 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.160697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.160652 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:39:53.182600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.178705 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:53.203601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.194725 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.203601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.194952 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.203601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.203336 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.355597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.329741 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:39:53.357378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.335726 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:39:53.361829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.361705 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 01 22:39:53.406290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.404689 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.446470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.446423 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.509157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.509122 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:39:53.537447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.537413 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.537731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.537703 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:53.548628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.548595 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:53.548864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.548837 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:53.607235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.605285 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.607235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.606071 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:53.645653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.632738 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:53.645653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.632798 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.709958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.679562 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:39:53.709958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.701211 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:39:53.763644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.763551 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:53.812793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.812768 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:39:53.814006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.812776 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:53.972896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.972797 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:39:54.058616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:53.994691 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:39:54.058616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.012667 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.113244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.113213 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:39:54.131796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.131490 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.133693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.133308 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:39:54.138106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.137184 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:54.204858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.204814 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:39:54.209993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.205934 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.209993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.206179 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:39:54.222823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.222342 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:39:54.270803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.264515 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:39:54.270803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.265018 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:39:54.544977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.544885 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:39:54.559514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.559489 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:54.563392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.563344 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.563521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.563412 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 01 22:39:54.563869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.563840 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 22:39:54.563966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.563882 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 22:39:54.616784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.616751 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:54.686605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.686136 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:54.686605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.686298 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:54.824304 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.824221 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:54.824560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:54.824520 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:55.028364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.028333 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:39:55.078476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.077202 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:39:55.354438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.351060 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:55.394631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.374381 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:39:55.769845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.769800 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:55.772011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.771983 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 22:39:55.783599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.782796 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:55.785621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.784978 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 22:39:55.968427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.967164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:55.968427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.968238 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:39:55.968427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:55.968286 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:39:56.181471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.181438 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:56.186808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.186732 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 22:39:56.300155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.300111 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk"] Nov 01 22:39:56.307825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.307797 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.318662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.312678 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:39:56.318662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.312724 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:39:56.318662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.313366 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:39:56.318662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.313437 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:39:56.319345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.319320 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk"] Nov 01 22:39:56.359767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.359728 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.359961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.359873 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728cf\" (UniqueName: \"kubernetes.io/projected/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-kube-api-access-728cf\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.360028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.359961 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.461315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.461241 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.461499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.461478 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.461777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.461660 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728cf\" (UniqueName: \"kubernetes.io/projected/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-kube-api-access-728cf\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.462330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.462307 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.463784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.463768 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.470298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.470272 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-728cf\" (UniqueName: \"kubernetes.io/projected/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-kube-api-access-728cf\") pod \"maas-api-revoke-sub-8d76e06a-v5skk\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:56.645428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:56.645399 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:57.004602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:57.004397 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk"] Nov 01 22:39:57.015498 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:57.015434 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6a6046_9ea5_4564_ab01_02edf3ddcdb5.slice/crio-bf7e7bcc2d54806d873a0b4c1fd534a9467abd4f8d57c04582ac1ee6eae359f3 WatchSource:0}: Error finding container bf7e7bcc2d54806d873a0b4c1fd534a9467abd4f8d57c04582ac1ee6eae359f3: Status 404 returned error can't find the container with id bf7e7bcc2d54806d873a0b4c1fd534a9467abd4f8d57c04582ac1ee6eae359f3 Nov 01 22:39:57.117067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:57.117028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:57.299222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:57.299121 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" event={"ID":"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5","Type":"ContainerStarted","Data":"bf7e7bcc2d54806d873a0b4c1fd534a9467abd4f8d57c04582ac1ee6eae359f3"} Nov 01 22:39:58.307929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.307892 4981 generic.go:356] "Generic (PLEG): container finished" podID="ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" containerID="7e4b991cc285adf46708ef65eb6f0cc03c66789d14f5735984f077af105ffd66" exitCode=0 Nov 01 22:39:58.308301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.307934 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" event={"ID":"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5","Type":"ContainerDied","Data":"7e4b991cc285adf46708ef65eb6f0cc03c66789d14f5735984f077af105ffd66"} Nov 01 22:39:58.492764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.492732 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc"] Nov 01 22:39:58.497363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.497343 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.510554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.510526 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc"] Nov 01 22:39:58.618704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.618671 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:58.625289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.625257 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dwb\" (UniqueName: \"kubernetes.io/projected/7e66f661-71b2-4950-a9fc-ed1c9532454a-kube-api-access-j2dwb\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.625433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.625418 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.625565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.625553 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.727693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.727652 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dwb\" (UniqueName: \"kubernetes.io/projected/7e66f661-71b2-4950-a9fc-ed1c9532454a-kube-api-access-j2dwb\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.727873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.727793 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.727929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.727914 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.728473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.728446 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.730307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.730288 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.736907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.736881 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dwb\" (UniqueName: \"kubernetes.io/projected/7e66f661-71b2-4950-a9fc-ed1c9532454a-kube-api-access-j2dwb\") pod \"maas-api-revoke-sub-2545daf8-jpvsc\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:58.818395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:58.818305 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:39:59.116442 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:39:59.116411 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e66f661_71b2_4950_a9fc_ed1c9532454a.slice/crio-ec3c22b8297ac0cede0298785fbe749536510f4f1a8eceed68484a65b7501b48 WatchSource:0}: Error finding container ec3c22b8297ac0cede0298785fbe749536510f4f1a8eceed68484a65b7501b48: Status 404 returned error can't find the container with id ec3c22b8297ac0cede0298785fbe749536510f4f1a8eceed68484a65b7501b48 Nov 01 22:39:59.117456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.117430 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc"] Nov 01 22:39:59.317004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.316971 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" event={"ID":"7e66f661-71b2-4950-a9fc-ed1c9532454a","Type":"ContainerStarted","Data":"ec3c22b8297ac0cede0298785fbe749536510f4f1a8eceed68484a65b7501b48"} Nov 01 22:39:59.461978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.461954 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:39:59.540393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.540364 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-openshift-ca\") pod \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " Nov 01 22:39:59.540591 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.540462 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-kube-api-access-728cf\" (UniqueName: \"kubernetes.io/projected/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-kube-api-access-728cf\") pod \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " Nov 01 22:39:59.540854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.540828 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-openshift-ca" pod "ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" (UID: "ee6a6046-9ea5-4564-ab01-02edf3ddcdb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:39:59.540944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.540885 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-xks-ca\") pod \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\" (UID: \"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5\") " Nov 01 22:39:59.541453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.541429 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:59.542763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.542713 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-kube-api-access-728cf" pod "ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" (UID: "ee6a6046-9ea5-4564-ab01-02edf3ddcdb5"). InnerVolumeSpecName "kube-api-access-728cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:39:59.542853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.542832 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-xks-ca" pod "ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" (UID: "ee6a6046-9ea5-4564-ab01-02edf3ddcdb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:39:59.545901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.545878 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:39:59.643098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.643005 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:39:59.643098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:39:59.643048 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-728cf\" (UniqueName: \"kubernetes.io/projected/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5-kube-api-access-728cf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:00.327791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.327723 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" Nov 01 22:40:00.327791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.327770 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk" event={"ID":"ee6a6046-9ea5-4564-ab01-02edf3ddcdb5","Type":"ContainerDied","Data":"bf7e7bcc2d54806d873a0b4c1fd534a9467abd4f8d57c04582ac1ee6eae359f3"} Nov 01 22:40:00.328213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.327822 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7e7bcc2d54806d873a0b4c1fd534a9467abd4f8d57c04582ac1ee6eae359f3" Nov 01 22:40:00.329513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.329489 4981 generic.go:356] "Generic (PLEG): container finished" podID="7e66f661-71b2-4950-a9fc-ed1c9532454a" containerID="5e42c57b3be3bb3b46adb2354057517e98c01e129d5834258345248e6f201db5" exitCode=0 Nov 01 22:40:00.329664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.329586 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" event={"ID":"7e66f661-71b2-4950-a9fc-ed1c9532454a","Type":"ContainerDied","Data":"5e42c57b3be3bb3b46adb2354057517e98c01e129d5834258345248e6f201db5"} Nov 01 22:40:00.477962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.477928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:00.854269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.854236 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j"] Nov 01 22:40:00.855018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.855003 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" containerName="revoke-keys" Nov 01 22:40:00.855077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.855020 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" containerName="revoke-keys" Nov 01 22:40:00.855179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.855168 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" containerName="revoke-keys" Nov 01 22:40:00.859186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.859164 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:00.869629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.869597 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j"] Nov 01 22:40:00.966116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.966080 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jmm\" (UniqueName: \"kubernetes.io/projected/c13b87a9-110f-44ac-9f70-a7348c6723df-kube-api-access-52jmm\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:00.966116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.966118 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:00.966423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:00.966408 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.028358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.028318 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n"] Nov 01 22:40:01.033181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.033155 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.041039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.041009 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n"] Nov 01 22:40:01.070501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.070452 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jmm\" (UniqueName: \"kubernetes.io/projected/c13b87a9-110f-44ac-9f70-a7348c6723df-kube-api-access-52jmm\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.070694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.070522 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.070809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.070785 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.072127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.072093 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.074010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.073978 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.079946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.079917 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jmm\" (UniqueName: \"kubernetes.io/projected/c13b87a9-110f-44ac-9f70-a7348c6723df-kube-api-access-52jmm\") pod \"maas-api-revoke-sub-2ad0fe7f-2v25j\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.172314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.172261 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.172314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.172300 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.172617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.172556 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t66w\" (UniqueName: \"kubernetes.io/projected/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-kube-api-access-9t66w\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.195126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.195094 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:01.275151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.275111 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.275330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.275166 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.275390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.275356 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t66w\" (UniqueName: \"kubernetes.io/projected/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-kube-api-access-9t66w\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.276160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.276132 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.278343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.278317 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.284699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.284670 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t66w\" (UniqueName: \"kubernetes.io/projected/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-kube-api-access-9t66w\") pod \"maas-api-revoke-sub-38563234-6ms2n\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.369608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.369336 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:01.446233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.443508 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:01.560203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.559936 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j"] Nov 01 22:40:01.579988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.579953 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:40:01.689415 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:01.689373 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e53eb8_52ca_49dd_b0ad_0dd1f5da30b7.slice/crio-6091864297f90cd69fbbf483848c8dbdbd90eeb085d455294aa43f553807a0c7 WatchSource:0}: Error finding container 6091864297f90cd69fbbf483848c8dbdbd90eeb085d455294aa43f553807a0c7: Status 404 returned error can't find the container with id 6091864297f90cd69fbbf483848c8dbdbd90eeb085d455294aa43f553807a0c7 Nov 01 22:40:01.690207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.690181 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n"] Nov 01 22:40:01.727690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.727656 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-xks-ca\") pod \"7e66f661-71b2-4950-a9fc-ed1c9532454a\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " Nov 01 22:40:01.728120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.728093 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e66f661-71b2-4950-a9fc-ed1c9532454a-kube-api-access-j2dwb\" (UniqueName: \"kubernetes.io/projected/7e66f661-71b2-4950-a9fc-ed1c9532454a-kube-api-access-j2dwb\") pod \"7e66f661-71b2-4950-a9fc-ed1c9532454a\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " Nov 01 22:40:01.728242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.728227 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-openshift-ca\") pod \"7e66f661-71b2-4950-a9fc-ed1c9532454a\" (UID: \"7e66f661-71b2-4950-a9fc-ed1c9532454a\") " Nov 01 22:40:01.728693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.728668 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-openshift-ca" pod "7e66f661-71b2-4950-a9fc-ed1c9532454a" (UID: "7e66f661-71b2-4950-a9fc-ed1c9532454a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:01.728919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.728896 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:01.730034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.730013 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-xks-ca" pod "7e66f661-71b2-4950-a9fc-ed1c9532454a" (UID: "7e66f661-71b2-4950-a9fc-ed1c9532454a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:01.730231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.730212 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e66f661-71b2-4950-a9fc-ed1c9532454a-kube-api-access-j2dwb" pod "7e66f661-71b2-4950-a9fc-ed1c9532454a" (UID: "7e66f661-71b2-4950-a9fc-ed1c9532454a"). InnerVolumeSpecName "kube-api-access-j2dwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:01.830440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.830399 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e66f661-71b2-4950-a9fc-ed1c9532454a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:01.830440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:01.830438 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dwb\" (UniqueName: \"kubernetes.io/projected/7e66f661-71b2-4950-a9fc-ed1c9532454a-kube-api-access-j2dwb\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:02.252858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.252775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:02.355175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.355139 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" event={"ID":"c13b87a9-110f-44ac-9f70-a7348c6723df","Type":"ContainerStarted","Data":"bc5d5541b0f3af08b9c26f5c90ad35daaa04e6a4de9495c4b1209412466af019"} Nov 01 22:40:02.355378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.355199 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" event={"ID":"c13b87a9-110f-44ac-9f70-a7348c6723df","Type":"ContainerStarted","Data":"5a97a309a23f75dce11e2f96178c53e616ea39ab9592c57b26aea1577e44c9fc"} Nov 01 22:40:02.357800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.357771 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" Nov 01 22:40:02.357963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.357809 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc" event={"ID":"7e66f661-71b2-4950-a9fc-ed1c9532454a","Type":"ContainerDied","Data":"ec3c22b8297ac0cede0298785fbe749536510f4f1a8eceed68484a65b7501b48"} Nov 01 22:40:02.357963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.357876 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec3c22b8297ac0cede0298785fbe749536510f4f1a8eceed68484a65b7501b48" Nov 01 22:40:02.360239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.360205 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" event={"ID":"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7","Type":"ContainerStarted","Data":"ae8557051f27b94b73af6b87311a0bec0ce0f4b2b58d793235885b691c77eb4d"} Nov 01 22:40:02.360382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.360254 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" event={"ID":"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7","Type":"ContainerStarted","Data":"6091864297f90cd69fbbf483848c8dbdbd90eeb085d455294aa43f553807a0c7"} Nov 01 22:40:02.384598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.379003 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" podStartSLOduration=2.378983964 podStartE2EDuration="2.378983964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:02.374760569 +0000 UTC m=+967.802638501" watchObservedRunningTime="2026-11-01 22:40:02.378983964 +0000 UTC m=+967.806861902" Nov 01 22:40:02.405007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:02.404953 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" podStartSLOduration=1.404938939 podStartE2EDuration="1.404938939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:02.400705844 +0000 UTC m=+967.828583776" watchObservedRunningTime="2026-11-01 22:40:02.404938939 +0000 UTC m=+967.832816870" Nov 01 22:40:03.169871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:03.169831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:03.384822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:03.384782 4981 generic.go:356] "Generic (PLEG): container finished" podID="c13b87a9-110f-44ac-9f70-a7348c6723df" containerID="bc5d5541b0f3af08b9c26f5c90ad35daaa04e6a4de9495c4b1209412466af019" exitCode=0 Nov 01 22:40:03.385204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:03.384864 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" event={"ID":"c13b87a9-110f-44ac-9f70-a7348c6723df","Type":"ContainerDied","Data":"bc5d5541b0f3af08b9c26f5c90ad35daaa04e6a4de9495c4b1209412466af019"} Nov 01 22:40:03.386751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:03.386729 4981 generic.go:356] "Generic (PLEG): container finished" podID="50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" containerID="ae8557051f27b94b73af6b87311a0bec0ce0f4b2b58d793235885b691c77eb4d" exitCode=0 Nov 01 22:40:03.386868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:03.386758 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" event={"ID":"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7","Type":"ContainerDied","Data":"ae8557051f27b94b73af6b87311a0bec0ce0f4b2b58d793235885b691c77eb4d"} Nov 01 22:40:03.896406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:03.896373 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:04.603031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.603001 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:04.641719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.641685 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:04.671617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.671517 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:04.682743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.682713 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-openshift-ca\") pod \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " Nov 01 22:40:04.682952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.682927 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-openshift-ca\") pod \"c13b87a9-110f-44ac-9f70-a7348c6723df\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " Nov 01 22:40:04.683054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.683020 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-kube-api-access-9t66w\" (UniqueName: \"kubernetes.io/projected/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-kube-api-access-9t66w\") pod \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " Nov 01 22:40:04.683156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.683138 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13b87a9-110f-44ac-9f70-a7348c6723df-kube-api-access-52jmm\" (UniqueName: \"kubernetes.io/projected/c13b87a9-110f-44ac-9f70-a7348c6723df-kube-api-access-52jmm\") pod \"c13b87a9-110f-44ac-9f70-a7348c6723df\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " Nov 01 22:40:04.683216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.683181 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-openshift-ca" pod "50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" (UID: "50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:04.683271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.683254 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-openshift-ca" pod "c13b87a9-110f-44ac-9f70-a7348c6723df" (UID: "c13b87a9-110f-44ac-9f70-a7348c6723df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:04.683325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.683276 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-xks-ca\") pod \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\" (UID: \"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7\") " Nov 01 22:40:04.683385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.683367 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-xks-ca\") pod \"c13b87a9-110f-44ac-9f70-a7348c6723df\" (UID: \"c13b87a9-110f-44ac-9f70-a7348c6723df\") " Nov 01 22:40:04.684144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.684114 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:04.684144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.684145 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:04.686103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.686075 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-kube-api-access-9t66w" pod "50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" (UID: "50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7"). InnerVolumeSpecName "kube-api-access-9t66w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:04.686213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.686122 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-xks-ca" pod "50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" (UID: "50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:04.686383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.686363 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13b87a9-110f-44ac-9f70-a7348c6723df-kube-api-access-52jmm" pod "c13b87a9-110f-44ac-9f70-a7348c6723df" (UID: "c13b87a9-110f-44ac-9f70-a7348c6723df"). InnerVolumeSpecName "kube-api-access-52jmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:04.690098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.690077 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-xks-ca" pod "c13b87a9-110f-44ac-9f70-a7348c6723df" (UID: "c13b87a9-110f-44ac-9f70-a7348c6723df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:04.780033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.780000 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7"] Nov 01 22:40:04.780754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.780733 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7e66f661-71b2-4950-a9fc-ed1c9532454a" containerName="revoke-keys" Nov 01 22:40:04.780754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.780751 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e66f661-71b2-4950-a9fc-ed1c9532454a" containerName="revoke-keys" Nov 01 22:40:04.780949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.780801 4981 cpu_manager.go:414] "containerMap: removing container" podUID="50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" containerName="revoke-keys" Nov 01 22:40:04.780949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.780809 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" containerName="revoke-keys" Nov 01 22:40:04.780949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.780840 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c13b87a9-110f-44ac-9f70-a7348c6723df" containerName="revoke-keys" Nov 01 22:40:04.780949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.780845 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13b87a9-110f-44ac-9f70-a7348c6723df" containerName="revoke-keys" Nov 01 22:40:04.781162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.781044 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e66f661-71b2-4950-a9fc-ed1c9532454a" containerName="revoke-keys" Nov 01 22:40:04.781162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.781069 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" containerName="revoke-keys" Nov 01 22:40:04.781162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.781083 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13b87a9-110f-44ac-9f70-a7348c6723df" containerName="revoke-keys" Nov 01 22:40:04.787305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.787268 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:04.787305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.787305 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13b87a9-110f-44ac-9f70-a7348c6723df-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:04.787632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.787319 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t66w\" (UniqueName: \"kubernetes.io/projected/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7-kube-api-access-9t66w\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:04.787632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.787334 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jmm\" (UniqueName: \"kubernetes.io/projected/c13b87a9-110f-44ac-9f70-a7348c6723df-kube-api-access-52jmm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:04.790326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.790293 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7"] Nov 01 22:40:04.790602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.790549 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.888890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.888851 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69bzm\" (UniqueName: \"kubernetes.io/projected/c509abba-f1b5-4442-8e6d-c762fce969d0-kube-api-access-69bzm\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.889067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.889007 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.889150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.889068 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.990435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.990402 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69bzm\" (UniqueName: \"kubernetes.io/projected/c509abba-f1b5-4442-8e6d-c762fce969d0-kube-api-access-69bzm\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.990719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.990482 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.990719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.990545 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.991271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.991246 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.992873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.992854 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:04.999544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:04.999520 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69bzm\" (UniqueName: \"kubernetes.io/projected/c509abba-f1b5-4442-8e6d-c762fce969d0-kube-api-access-69bzm\") pod \"maas-api-revoke-sub-f0a3c6ae-bv4s7\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:05.120315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.120281 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:05.404345 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:05.404311 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc509abba_f1b5_4442_8e6d_c762fce969d0.slice/crio-eb609c5a9d007467af7bf931d21a0082cf96f692a750269090ea85a429c9841b WatchSource:0}: Error finding container eb609c5a9d007467af7bf931d21a0082cf96f692a750269090ea85a429c9841b: Status 404 returned error can't find the container with id eb609c5a9d007467af7bf931d21a0082cf96f692a750269090ea85a429c9841b Nov 01 22:40:05.405022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.404995 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7"] Nov 01 22:40:05.405903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.405882 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" Nov 01 22:40:05.406019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.405904 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j" event={"ID":"c13b87a9-110f-44ac-9f70-a7348c6723df","Type":"ContainerDied","Data":"5a97a309a23f75dce11e2f96178c53e616ea39ab9592c57b26aea1577e44c9fc"} Nov 01 22:40:05.406019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.405956 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a97a309a23f75dce11e2f96178c53e616ea39ab9592c57b26aea1577e44c9fc" Nov 01 22:40:05.407693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.407675 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" Nov 01 22:40:05.407765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.407713 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n" event={"ID":"50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7","Type":"ContainerDied","Data":"6091864297f90cd69fbbf483848c8dbdbd90eeb085d455294aa43f553807a0c7"} Nov 01 22:40:05.407765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.407746 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6091864297f90cd69fbbf483848c8dbdbd90eeb085d455294aa43f553807a0c7" Nov 01 22:40:05.493803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.493763 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt"] Nov 01 22:40:05.501104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.501077 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.508863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.507771 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt"] Nov 01 22:40:05.577937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.577903 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:05.605642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.605608 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.605984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.605662 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtlqs\" (UniqueName: \"kubernetes.io/projected/d0c49ffc-13c2-413e-8618-b335f3a49890-kube-api-access-xtlqs\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.605984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.605706 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.707108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.707059 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.707331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.707132 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtlqs\" (UniqueName: \"kubernetes.io/projected/d0c49ffc-13c2-413e-8618-b335f3a49890-kube-api-access-xtlqs\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.707331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.707196 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.707876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.707852 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.709692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.709668 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.716717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.716693 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtlqs\" (UniqueName: \"kubernetes.io/projected/d0c49ffc-13c2-413e-8618-b335f3a49890-kube-api-access-xtlqs\") pod \"maas-api-revoke-sub-2b3d6ecc-txzgt\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:05.822003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:05.821970 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:06.205980 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:06.205948 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c49ffc_13c2_413e_8618_b335f3a49890.slice/crio-0ca2f7e5d09cfc9b55a90c6d1a1f78551716506658d2fb0a0228646df746661b WatchSource:0}: Error finding container 0ca2f7e5d09cfc9b55a90c6d1a1f78551716506658d2fb0a0228646df746661b: Status 404 returned error can't find the container with id 0ca2f7e5d09cfc9b55a90c6d1a1f78551716506658d2fb0a0228646df746661b Nov 01 22:40:06.209005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:06.208981 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt"] Nov 01 22:40:06.418703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:06.418664 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" event={"ID":"d0c49ffc-13c2-413e-8618-b335f3a49890","Type":"ContainerStarted","Data":"0ca2f7e5d09cfc9b55a90c6d1a1f78551716506658d2fb0a0228646df746661b"} Nov 01 22:40:06.420455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:06.420426 4981 generic.go:356] "Generic (PLEG): container finished" podID="c509abba-f1b5-4442-8e6d-c762fce969d0" containerID="402042e1d0c3173d8cb1d7e1d84db4da3c7e16f5e028eceb98f31681d63ec140" exitCode=0 Nov 01 22:40:06.420563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:06.420468 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" event={"ID":"c509abba-f1b5-4442-8e6d-c762fce969d0","Type":"ContainerDied","Data":"402042e1d0c3173d8cb1d7e1d84db4da3c7e16f5e028eceb98f31681d63ec140"} Nov 01 22:40:06.420563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:06.420508 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" event={"ID":"c509abba-f1b5-4442-8e6d-c762fce969d0","Type":"ContainerStarted","Data":"eb609c5a9d007467af7bf931d21a0082cf96f692a750269090ea85a429c9841b"} Nov 01 22:40:06.450286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:06.450255 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:07.332504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.332470 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:07.434071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.434034 4981 generic.go:356] "Generic (PLEG): container finished" podID="d0c49ffc-13c2-413e-8618-b335f3a49890" containerID="29028ff8399b45af117f22fdc5c5148c142891f9e991dfd658c0e191360b62d7" exitCode=0 Nov 01 22:40:07.434333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.434274 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" event={"ID":"d0c49ffc-13c2-413e-8618-b335f3a49890","Type":"ContainerDied","Data":"29028ff8399b45af117f22fdc5c5148c142891f9e991dfd658c0e191360b62d7"} Nov 01 22:40:07.440847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.440820 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92"] Nov 01 22:40:07.445866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.445836 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.491351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.491314 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92"] Nov 01 22:40:07.541831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.541460 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4dj\" (UniqueName: \"kubernetes.io/projected/b3d97c14-60db-45b5-ac17-13d5e9a2293e-kube-api-access-5q4dj\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.541831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.541742 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.541831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.541799 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.644503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.644470 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.644638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.644601 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.644874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.644853 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4dj\" (UniqueName: \"kubernetes.io/projected/b3d97c14-60db-45b5-ac17-13d5e9a2293e-kube-api-access-5q4dj\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.645219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.645195 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.647093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.647062 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.655615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.655588 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4dj\" (UniqueName: \"kubernetes.io/projected/b3d97c14-60db-45b5-ac17-13d5e9a2293e-kube-api-access-5q4dj\") pod \"maas-api-revoke-sub-e34bc39a-cxf92\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.682083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.682057 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:07.745813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.745728 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c509abba-f1b5-4442-8e6d-c762fce969d0-kube-api-access-69bzm\" (UniqueName: \"kubernetes.io/projected/c509abba-f1b5-4442-8e6d-c762fce969d0-kube-api-access-69bzm\") pod \"c509abba-f1b5-4442-8e6d-c762fce969d0\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " Nov 01 22:40:07.745967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.745951 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-xks-ca\") pod \"c509abba-f1b5-4442-8e6d-c762fce969d0\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " Nov 01 22:40:07.746125 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.746105 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-openshift-ca\") pod \"c509abba-f1b5-4442-8e6d-c762fce969d0\" (UID: \"c509abba-f1b5-4442-8e6d-c762fce969d0\") " Nov 01 22:40:07.746466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.746441 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-openshift-ca" pod "c509abba-f1b5-4442-8e6d-c762fce969d0" (UID: "c509abba-f1b5-4442-8e6d-c762fce969d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:07.746952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.746927 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:07.748583 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.748555 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-xks-ca" pod "c509abba-f1b5-4442-8e6d-c762fce969d0" (UID: "c509abba-f1b5-4442-8e6d-c762fce969d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:07.748914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.748895 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c509abba-f1b5-4442-8e6d-c762fce969d0-kube-api-access-69bzm" pod "c509abba-f1b5-4442-8e6d-c762fce969d0" (UID: "c509abba-f1b5-4442-8e6d-c762fce969d0"). InnerVolumeSpecName "kube-api-access-69bzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:07.800312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.800280 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:07.849147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.849116 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69bzm\" (UniqueName: \"kubernetes.io/projected/c509abba-f1b5-4442-8e6d-c762fce969d0-kube-api-access-69bzm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:07.849147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:07.849145 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c509abba-f1b5-4442-8e6d-c762fce969d0-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:08.093736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.092478 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92"] Nov 01 22:40:08.097745 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:08.097705 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d97c14_60db_45b5_ac17_13d5e9a2293e.slice/crio-a853634b41df01118a84230b5074db5be489beb8275647ce5be45c6e3009ea2f WatchSource:0}: Error finding container a853634b41df01118a84230b5074db5be489beb8275647ce5be45c6e3009ea2f: Status 404 returned error can't find the container with id a853634b41df01118a84230b5074db5be489beb8275647ce5be45c6e3009ea2f Nov 01 22:40:08.248904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.248872 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:08.444037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.444004 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" Nov 01 22:40:08.444462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.444008 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7" event={"ID":"c509abba-f1b5-4442-8e6d-c762fce969d0","Type":"ContainerDied","Data":"eb609c5a9d007467af7bf931d21a0082cf96f692a750269090ea85a429c9841b"} Nov 01 22:40:08.444462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.444148 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb609c5a9d007467af7bf931d21a0082cf96f692a750269090ea85a429c9841b" Nov 01 22:40:08.445505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.445475 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" event={"ID":"b3d97c14-60db-45b5-ac17-13d5e9a2293e","Type":"ContainerStarted","Data":"a853634b41df01118a84230b5074db5be489beb8275647ce5be45c6e3009ea2f"} Nov 01 22:40:08.602612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.602586 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:08.664395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.664368 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-openshift-ca\") pod \"d0c49ffc-13c2-413e-8618-b335f3a49890\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " Nov 01 22:40:08.664553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.664542 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-xks-ca\") pod \"d0c49ffc-13c2-413e-8618-b335f3a49890\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " Nov 01 22:40:08.664718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.664704 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c49ffc-13c2-413e-8618-b335f3a49890-kube-api-access-xtlqs\" (UniqueName: \"kubernetes.io/projected/d0c49ffc-13c2-413e-8618-b335f3a49890-kube-api-access-xtlqs\") pod \"d0c49ffc-13c2-413e-8618-b335f3a49890\" (UID: \"d0c49ffc-13c2-413e-8618-b335f3a49890\") " Nov 01 22:40:08.664841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.664815 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-openshift-ca" pod "d0c49ffc-13c2-413e-8618-b335f3a49890" (UID: "d0c49ffc-13c2-413e-8618-b335f3a49890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:08.665273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.665251 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:08.666791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.666770 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-xks-ca" pod "d0c49ffc-13c2-413e-8618-b335f3a49890" (UID: "d0c49ffc-13c2-413e-8618-b335f3a49890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:08.666878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.666841 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c49ffc-13c2-413e-8618-b335f3a49890-kube-api-access-xtlqs" pod "d0c49ffc-13c2-413e-8618-b335f3a49890" (UID: "d0c49ffc-13c2-413e-8618-b335f3a49890"). InnerVolumeSpecName "kube-api-access-xtlqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:08.767385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.767355 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c49ffc-13c2-413e-8618-b335f3a49890-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:08.767385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:08.767384 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtlqs\" (UniqueName: \"kubernetes.io/projected/d0c49ffc-13c2-413e-8618-b335f3a49890-kube-api-access-xtlqs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:09.087650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:09.087558 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:09.200643 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:40:09.200611 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d97c14_60db_45b5_ac17_13d5e9a2293e.slice/crio-conmon-b26a48f87d171e62c1ce8c9de92eea214b135575367f3349d4e6ea169954b049.scope\": RecentStats: unable to find data in memory cache]" Nov 01 22:40:09.457870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:09.457833 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" Nov 01 22:40:09.458271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:09.457877 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt" event={"ID":"d0c49ffc-13c2-413e-8618-b335f3a49890","Type":"ContainerDied","Data":"0ca2f7e5d09cfc9b55a90c6d1a1f78551716506658d2fb0a0228646df746661b"} Nov 01 22:40:09.458271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:09.457932 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca2f7e5d09cfc9b55a90c6d1a1f78551716506658d2fb0a0228646df746661b" Nov 01 22:40:09.459829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:09.459801 4981 generic.go:356] "Generic (PLEG): container finished" podID="b3d97c14-60db-45b5-ac17-13d5e9a2293e" containerID="b26a48f87d171e62c1ce8c9de92eea214b135575367f3349d4e6ea169954b049" exitCode=0 Nov 01 22:40:09.459957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:09.459887 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" event={"ID":"b3d97c14-60db-45b5-ac17-13d5e9a2293e","Type":"ContainerDied","Data":"b26a48f87d171e62c1ce8c9de92eea214b135575367f3349d4e6ea169954b049"} Nov 01 22:40:09.769282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:09.769206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:10.583180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.583149 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:10.656829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.656799 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:10.805587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.805493 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-openshift-ca\") pod \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " Nov 01 22:40:10.805743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.805665 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-xks-ca\") pod \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " Nov 01 22:40:10.805789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.805768 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d97c14-60db-45b5-ac17-13d5e9a2293e-kube-api-access-5q4dj\" (UniqueName: \"kubernetes.io/projected/b3d97c14-60db-45b5-ac17-13d5e9a2293e-kube-api-access-5q4dj\") pod \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\" (UID: \"b3d97c14-60db-45b5-ac17-13d5e9a2293e\") " Nov 01 22:40:10.806002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.805976 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-openshift-ca" pod "b3d97c14-60db-45b5-ac17-13d5e9a2293e" (UID: "b3d97c14-60db-45b5-ac17-13d5e9a2293e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:10.806528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.806505 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:10.807860 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.807830 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-xks-ca" pod "b3d97c14-60db-45b5-ac17-13d5e9a2293e" (UID: "b3d97c14-60db-45b5-ac17-13d5e9a2293e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:10.807965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.807947 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d97c14-60db-45b5-ac17-13d5e9a2293e-kube-api-access-5q4dj" pod "b3d97c14-60db-45b5-ac17-13d5e9a2293e" (UID: "b3d97c14-60db-45b5-ac17-13d5e9a2293e"). InnerVolumeSpecName "kube-api-access-5q4dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:10.908234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.908199 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3d97c14-60db-45b5-ac17-13d5e9a2293e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:10.908234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:10.908231 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4dj\" (UniqueName: \"kubernetes.io/projected/b3d97c14-60db-45b5-ac17-13d5e9a2293e-kube-api-access-5q4dj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:11.357431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:11.357401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:11.482558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:11.482524 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" Nov 01 22:40:11.482737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:11.482566 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92" event={"ID":"b3d97c14-60db-45b5-ac17-13d5e9a2293e","Type":"ContainerDied","Data":"a853634b41df01118a84230b5074db5be489beb8275647ce5be45c6e3009ea2f"} Nov 01 22:40:11.482737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:11.482686 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a853634b41df01118a84230b5074db5be489beb8275647ce5be45c6e3009ea2f" Nov 01 22:40:12.136977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:12.136936 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:12.866352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:12.866318 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:12.976658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:12.976564 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:14.750727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.750678 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.826923 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:40:14.826888 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:40:14.835605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.832443 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.866660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.866611 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.867351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.866951 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 01 22:40:14.867351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.867054 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.871648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.871612 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:40:14.872391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.872231 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.877818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.877792 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.884384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.884353 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.884848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.884822 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.928599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.928306 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:14.946600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:14.943650 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.064659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.063669 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.189599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.189104 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.348738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.347665 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.516607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.467456 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:40:15.516607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.493674 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.586625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.586567 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:15.586811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.586723 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.610284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.610007 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.749603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.734295 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:15.858163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.858116 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.859386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.859361 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:40:15.872761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.872666 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.876828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.872986 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:40:15.895601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.874829 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.897099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.877233 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:40:15.897516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.897496 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:40:15.898684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.898667 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:40:15.921949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.893227 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:40:15.922499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.898557 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.922667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.890677 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:40:15.923368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.923345 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:40:15.923530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.923518 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:40:15.924519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.893199 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:15.925802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.925784 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:40:15.943182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.943143 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:40:15.943898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:15.943877 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:40:16.355911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.355480 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:40:16.379449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.377923 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:40:16.417667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.396165 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:40:16.417667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.397603 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:40:16.471644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.468735 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:16.506640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.475090 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded" Nov 01 22:40:16.506640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.475356 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:40:16.506640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.487729 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.506640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.491082 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:40:16.506640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.494828 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:40:16.506640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.504002 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:40:16.540191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.540152 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:16.785601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.767863 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.785601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.768122 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:16.785601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.768158 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.811601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.808796 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:40:16.918603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.906449 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:40:16.976635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.932692 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:16.976635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.938777 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.976635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.940033 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:40:16.976635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.941057 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:16.976635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.973837 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:40:16.976635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.974152 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:40:17.011835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:16.986757 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:40:17.034848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.026676 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.034848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.027223 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.071653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.053646 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.240857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.240792 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:17.241242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.241196 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.406391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.380688 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.426601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.410672 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.491237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.491196 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.768743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.493912 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.793653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.519751 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.830636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.823891 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.830962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.569739 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:17.831280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.831155 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:17.981615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.726950 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:18.038668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.946678 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:40:18.038668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.965690 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.038668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.989692 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.038668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.989737 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:40:18.038668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.989756 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:40:18.039265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.039227 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.039509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.994025 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.039667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:17.990491 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:40:18.059892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.059831 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.062840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.062788 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.152664 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:40:18.148683 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.136s" Nov 01 22:40:18.152664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.148769 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:40:18.400955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.399703 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.507737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.477774 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.719116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.719081 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:18.798047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.772976 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.798424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.798388 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.858607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.851923 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:40:18.903912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.894038 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:18.934764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:18.719647 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.027924 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.035663 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.040700 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.040730 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.044920 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.083863 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.083899 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.083924 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:40:19.110650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.083955 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:40:19.369935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.369827 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.417744 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm"] Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419209 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c509abba-f1b5-4442-8e6d-c762fce969d0" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419231 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c509abba-f1b5-4442-8e6d-c762fce969d0" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419276 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b3d97c14-60db-45b5-ac17-13d5e9a2293e" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419285 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d97c14-60db-45b5-ac17-13d5e9a2293e" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419317 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d0c49ffc-13c2-413e-8618-b335f3a49890" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419339 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c49ffc-13c2-413e-8618-b335f3a49890" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419570 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d97c14-60db-45b5-ac17-13d5e9a2293e" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419606 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c49ffc-13c2-413e-8618-b335f3a49890" containerName="revoke-keys" Nov 01 22:40:19.428621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.419648 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c509abba-f1b5-4442-8e6d-c762fce969d0" containerName="revoke-keys" Nov 01 22:40:19.496922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.496880 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:40:19.668190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.575008 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:19.943015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.942972 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:40:19.996147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.996101 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:40:19.996407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:19.996378 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:20.017830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.017798 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:40:20.020071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.020047 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.039435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.039398 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:40:20.039696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.039677 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm"] Nov 01 22:40:20.040955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.040280 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:40:20.044368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.044338 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:40:20.044498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.044417 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:40:20.044498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.044459 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:40:20.046404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.046385 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:40:20.046678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.046649 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:40:20.051470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.051431 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:40:20.053818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.053798 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:40:20.067738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.067717 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:40:20.068515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.068496 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:40:20.069330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.069314 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:40:20.071229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.071205 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:40:20.071332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.071255 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:40:20.071332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.071289 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:40:20.071621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.071597 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:40:20.071854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.070617 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:40:20.072164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.070536 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:20.098475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.096870 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx"] Nov 01 22:40:20.105869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.105834 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.108771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.108752 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx"] Nov 01 22:40:20.183322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.183294 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r"] Nov 01 22:40:20.188981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.188957 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.190225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.190198 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r"] Nov 01 22:40:20.204542 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.204510 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lgnx\" (UniqueName: \"kubernetes.io/projected/c451365c-af96-46e5-8413-8e73372f7446-kube-api-access-6lgnx\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.205033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.205008 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.205332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.205309 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c451365c-af96-46e5-8413-8e73372f7446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.207962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.207938 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c451365c-af96-46e5-8413-8e73372f7446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.208066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.208030 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.208424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.208405 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdjf\" (UniqueName: \"kubernetes.io/projected/6db5039a-aa92-4889-b66e-187cbdaefdb7-kube-api-access-5fdjf\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.309615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309566 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lgnx\" (UniqueName: \"kubernetes.io/projected/c451365c-af96-46e5-8413-8e73372f7446-kube-api-access-6lgnx\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.309793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309674 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fqs\" (UniqueName: \"kubernetes.io/projected/337a224f-077a-4646-b0a3-ce2a7c1179a8-kube-api-access-p6fqs\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.309793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309722 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.309793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309756 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c451365c-af96-46e5-8413-8e73372f7446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.309793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309793 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c451365c-af96-46e5-8413-8e73372f7446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.309988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309810 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.309988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309884 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.309988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.309961 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.310122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.310009 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdjf\" (UniqueName: \"kubernetes.io/projected/6db5039a-aa92-4889-b66e-187cbdaefdb7-kube-api-access-5fdjf\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.310887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.310859 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c451365c-af96-46e5-8413-8e73372f7446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.314187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.312226 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.314187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.312585 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c451365c-af96-46e5-8413-8e73372f7446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.314795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.314770 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.340331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.337835 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdjf\" (UniqueName: \"kubernetes.io/projected/6db5039a-aa92-4889-b66e-187cbdaefdb7-kube-api-access-5fdjf\") pod \"maas-api-revoke-sub-9269d490-8tgjx\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.342999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.342964 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lgnx\" (UniqueName: \"kubernetes.io/projected/c451365c-af96-46e5-8413-8e73372f7446-kube-api-access-6lgnx\") pod \"maas-api-revoke-sub-bf9828c7-jkgwm\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.388226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.387991 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:20.412926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.412883 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.413105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.413024 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.413244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.413226 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fqs\" (UniqueName: \"kubernetes.io/projected/337a224f-077a-4646-b0a3-ce2a7c1179a8-kube-api-access-p6fqs\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.414335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.414306 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.420252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.420224 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.435517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.435488 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fqs\" (UniqueName: \"kubernetes.io/projected/337a224f-077a-4646-b0a3-ce2a7c1179a8-kube-api-access-p6fqs\") pod \"maas-api-revoke-sub-b0a51171-25g6r\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.436336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.436310 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:20.520660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.519071 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:20.766867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.766840 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm"] Nov 01 22:40:20.770137 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:20.770093 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db5039a_aa92_4889_b66e_187cbdaefdb7.slice/crio-93949c1e748ca48591aaf24d26f1b50acd0cd9efd1aab45a3cff1de6689da1b2 WatchSource:0}: Error finding container 93949c1e748ca48591aaf24d26f1b50acd0cd9efd1aab45a3cff1de6689da1b2: Status 404 returned error can't find the container with id 93949c1e748ca48591aaf24d26f1b50acd0cd9efd1aab45a3cff1de6689da1b2 Nov 01 22:40:20.772966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.772943 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx"] Nov 01 22:40:20.849788 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:20.849759 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337a224f_077a_4646_b0a3_ce2a7c1179a8.slice/crio-55049355c32eceaa626947a610143ef58ecc54b9541332a7ae6dbc65154fc252 WatchSource:0}: Error finding container 55049355c32eceaa626947a610143ef58ecc54b9541332a7ae6dbc65154fc252: Status 404 returned error can't find the container with id 55049355c32eceaa626947a610143ef58ecc54b9541332a7ae6dbc65154fc252 Nov 01 22:40:20.853655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:20.853633 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r"] Nov 01 22:40:21.081805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:21.081734 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" event={"ID":"c451365c-af96-46e5-8413-8e73372f7446","Type":"ContainerStarted","Data":"e2050c11c8c710a2775c7babdfb038bc2b496e24b83ee3954c40a3e5c26c97b5"} Nov 01 22:40:21.083411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:21.083384 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" event={"ID":"337a224f-077a-4646-b0a3-ce2a7c1179a8","Type":"ContainerStarted","Data":"55049355c32eceaa626947a610143ef58ecc54b9541332a7ae6dbc65154fc252"} Nov 01 22:40:21.084807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:21.084785 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" event={"ID":"6db5039a-aa92-4889-b66e-187cbdaefdb7","Type":"ContainerStarted","Data":"93949c1e748ca48591aaf24d26f1b50acd0cd9efd1aab45a3cff1de6689da1b2"} Nov 01 22:40:21.538476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:21.538310 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:22.096327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.096297 4981 generic.go:356] "Generic (PLEG): container finished" podID="337a224f-077a-4646-b0a3-ce2a7c1179a8" containerID="a823c75c01a7405288e9d77268b31f84db96dc0bac3d90a22b9bc58b8dc1ef08" exitCode=0 Nov 01 22:40:22.096495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.096331 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" event={"ID":"337a224f-077a-4646-b0a3-ce2a7c1179a8","Type":"ContainerDied","Data":"a823c75c01a7405288e9d77268b31f84db96dc0bac3d90a22b9bc58b8dc1ef08"} Nov 01 22:40:22.098123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.098105 4981 generic.go:356] "Generic (PLEG): container finished" podID="6db5039a-aa92-4889-b66e-187cbdaefdb7" containerID="277fa3098e5fdec4fb6cd51c295594f18d3de87cc11f84173ce34029baa6dee3" exitCode=0 Nov 01 22:40:22.098212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.098194 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" event={"ID":"6db5039a-aa92-4889-b66e-187cbdaefdb7","Type":"ContainerDied","Data":"277fa3098e5fdec4fb6cd51c295594f18d3de87cc11f84173ce34029baa6dee3"} Nov 01 22:40:22.100034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.100013 4981 generic.go:356] "Generic (PLEG): container finished" podID="c451365c-af96-46e5-8413-8e73372f7446" containerID="617d8f1f36fd1bfefc149432f0e3f9787453d6d62235458420a6a21dd7da262d" exitCode=0 Nov 01 22:40:22.100136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.100041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" event={"ID":"c451365c-af96-46e5-8413-8e73372f7446","Type":"ContainerDied","Data":"617d8f1f36fd1bfefc149432f0e3f9787453d6d62235458420a6a21dd7da262d"} Nov 01 22:40:22.373795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.373715 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:22.906437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.906406 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk"] Nov 01 22:40:22.911341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.911322 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:22.923839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.923808 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk"] Nov 01 22:40:22.965152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.965120 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:22.965301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.965165 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7t5\" (UniqueName: \"kubernetes.io/projected/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-kube-api-access-pf7t5\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:22.965380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:22.965355 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.067082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.067047 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.067239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.067087 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7t5\" (UniqueName: \"kubernetes.io/projected/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-kube-api-access-pf7t5\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.067326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.067309 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.067828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.067811 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.069732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.069710 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.075899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.075872 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7t5\" (UniqueName: \"kubernetes.io/projected/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-kube-api-access-pf7t5\") pod \"maas-api-revoke-sub-24888703-bmnkk\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.236177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.236144 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:23.273941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.270656 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:23.396279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.395822 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:23.413728 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.413446 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:23.430360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.430148 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:23.433880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.433860 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:23.487070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487024 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-xks-ca\") pod \"6db5039a-aa92-4889-b66e-187cbdaefdb7\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " Nov 01 22:40:23.487222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487144 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-openshift-ca\") pod \"6db5039a-aa92-4889-b66e-187cbdaefdb7\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " Nov 01 22:40:23.487222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487187 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db5039a-aa92-4889-b66e-187cbdaefdb7-kube-api-access-5fdjf\" (UniqueName: \"kubernetes.io/projected/6db5039a-aa92-4889-b66e-187cbdaefdb7-kube-api-access-5fdjf\") pod \"6db5039a-aa92-4889-b66e-187cbdaefdb7\" (UID: \"6db5039a-aa92-4889-b66e-187cbdaefdb7\") " Nov 01 22:40:23.487312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487232 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c451365c-af96-46e5-8413-8e73372f7446-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c451365c-af96-46e5-8413-8e73372f7446-maas-api-xks-ca\") pod \"c451365c-af96-46e5-8413-8e73372f7446\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " Nov 01 22:40:23.487352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487344 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-xks-ca\") pod \"337a224f-077a-4646-b0a3-ce2a7c1179a8\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " Nov 01 22:40:23.487736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487698 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-openshift-ca" pod "6db5039a-aa92-4889-b66e-187cbdaefdb7" (UID: "6db5039a-aa92-4889-b66e-187cbdaefdb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:23.487859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487763 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c451365c-af96-46e5-8413-8e73372f7446-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c451365c-af96-46e5-8413-8e73372f7446-maas-api-openshift-ca\") pod \"c451365c-af96-46e5-8413-8e73372f7446\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " Nov 01 22:40:23.487996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.487974 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-openshift-ca\") pod \"337a224f-077a-4646-b0a3-ce2a7c1179a8\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " Nov 01 22:40:23.488050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.488024 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c451365c-af96-46e5-8413-8e73372f7446-kube-api-access-6lgnx\" (UniqueName: \"kubernetes.io/projected/c451365c-af96-46e5-8413-8e73372f7446-kube-api-access-6lgnx\") pod \"c451365c-af96-46e5-8413-8e73372f7446\" (UID: \"c451365c-af96-46e5-8413-8e73372f7446\") " Nov 01 22:40:23.488107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.488067 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337a224f-077a-4646-b0a3-ce2a7c1179a8-kube-api-access-p6fqs\" (UniqueName: \"kubernetes.io/projected/337a224f-077a-4646-b0a3-ce2a7c1179a8-kube-api-access-p6fqs\") pod \"337a224f-077a-4646-b0a3-ce2a7c1179a8\" (UID: \"337a224f-077a-4646-b0a3-ce2a7c1179a8\") " Nov 01 22:40:23.488829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.488796 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c451365c-af96-46e5-8413-8e73372f7446-maas-api-openshift-ca" pod "c451365c-af96-46e5-8413-8e73372f7446" (UID: "c451365c-af96-46e5-8413-8e73372f7446"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:23.489186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.489162 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.489186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.489186 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c451365c-af96-46e5-8413-8e73372f7446-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.489858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.489824 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-xks-ca" pod "337a224f-077a-4646-b0a3-ce2a7c1179a8" (UID: "337a224f-077a-4646-b0a3-ce2a7c1179a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:23.490643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.490614 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-openshift-ca" pod "337a224f-077a-4646-b0a3-ce2a7c1179a8" (UID: "337a224f-077a-4646-b0a3-ce2a7c1179a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:23.491535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.491507 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-xks-ca" pod "6db5039a-aa92-4889-b66e-187cbdaefdb7" (UID: "6db5039a-aa92-4889-b66e-187cbdaefdb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:23.491934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.491881 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db5039a-aa92-4889-b66e-187cbdaefdb7-kube-api-access-5fdjf" pod "6db5039a-aa92-4889-b66e-187cbdaefdb7" (UID: "6db5039a-aa92-4889-b66e-187cbdaefdb7"). InnerVolumeSpecName "kube-api-access-5fdjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:23.492135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.492108 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c451365c-af96-46e5-8413-8e73372f7446-kube-api-access-6lgnx" pod "c451365c-af96-46e5-8413-8e73372f7446" (UID: "c451365c-af96-46e5-8413-8e73372f7446"). InnerVolumeSpecName "kube-api-access-6lgnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:23.493260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.493066 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c451365c-af96-46e5-8413-8e73372f7446-maas-api-xks-ca" pod "c451365c-af96-46e5-8413-8e73372f7446" (UID: "c451365c-af96-46e5-8413-8e73372f7446"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:23.493260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.493147 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337a224f-077a-4646-b0a3-ce2a7c1179a8-kube-api-access-p6fqs" pod "337a224f-077a-4646-b0a3-ce2a7c1179a8" (UID: "337a224f-077a-4646-b0a3-ce2a7c1179a8"). InnerVolumeSpecName "kube-api-access-p6fqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:23.543806 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.543775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:23.590587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.590549 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.590587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.590587 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/337a224f-077a-4646-b0a3-ce2a7c1179a8-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.590757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.590597 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lgnx\" (UniqueName: \"kubernetes.io/projected/c451365c-af96-46e5-8413-8e73372f7446-kube-api-access-6lgnx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.590757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.590606 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fqs\" (UniqueName: \"kubernetes.io/projected/337a224f-077a-4646-b0a3-ce2a7c1179a8-kube-api-access-p6fqs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.590757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.590617 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db5039a-aa92-4889-b66e-187cbdaefdb7-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.590757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.590626 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fdjf\" (UniqueName: \"kubernetes.io/projected/6db5039a-aa92-4889-b66e-187cbdaefdb7-kube-api-access-5fdjf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.590757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.590635 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c451365c-af96-46e5-8413-8e73372f7446-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:23.641782 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:23.641749 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0d0707_bf21_41c9_9eff_ea900c0b8a9c.slice/crio-3bcbd9e305edd978d8005da93543d6e691a0aff38d7ba0bab7b600a99dc3d6e3 WatchSource:0}: Error finding container 3bcbd9e305edd978d8005da93543d6e691a0aff38d7ba0bab7b600a99dc3d6e3: Status 404 returned error can't find the container with id 3bcbd9e305edd978d8005da93543d6e691a0aff38d7ba0bab7b600a99dc3d6e3 Nov 01 22:40:23.644288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.644262 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk"] Nov 01 22:40:23.846955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:23.846876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:24.124611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.124500 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" event={"ID":"337a224f-077a-4646-b0a3-ce2a7c1179a8","Type":"ContainerDied","Data":"55049355c32eceaa626947a610143ef58ecc54b9541332a7ae6dbc65154fc252"} Nov 01 22:40:24.124611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.124515 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r" Nov 01 22:40:24.125189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.124617 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55049355c32eceaa626947a610143ef58ecc54b9541332a7ae6dbc65154fc252" Nov 01 22:40:24.128279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.128229 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" event={"ID":"6db5039a-aa92-4889-b66e-187cbdaefdb7","Type":"ContainerDied","Data":"93949c1e748ca48591aaf24d26f1b50acd0cd9efd1aab45a3cff1de6689da1b2"} Nov 01 22:40:24.128405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.128299 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93949c1e748ca48591aaf24d26f1b50acd0cd9efd1aab45a3cff1de6689da1b2" Nov 01 22:40:24.128405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.128372 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx" Nov 01 22:40:24.130414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.130388 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" event={"ID":"c451365c-af96-46e5-8413-8e73372f7446","Type":"ContainerDied","Data":"e2050c11c8c710a2775c7babdfb038bc2b496e24b83ee3954c40a3e5c26c97b5"} Nov 01 22:40:24.130511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.130433 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2050c11c8c710a2775c7babdfb038bc2b496e24b83ee3954c40a3e5c26c97b5" Nov 01 22:40:24.130511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.130494 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm" Nov 01 22:40:24.140748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.140725 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" event={"ID":"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c","Type":"ContainerStarted","Data":"2d904167066cb89d67a11d063d27b4382b421e814871652eed6d10f2610472e0"} Nov 01 22:40:24.140854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.140767 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" event={"ID":"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c","Type":"ContainerStarted","Data":"3bcbd9e305edd978d8005da93543d6e691a0aff38d7ba0bab7b600a99dc3d6e3"} Nov 01 22:40:24.175809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.175762 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" podStartSLOduration=2.175748924 podStartE2EDuration="2.175748924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:24.166227406 +0000 UTC m=+989.594105340" watchObservedRunningTime="2026-11-01 22:40:24.175748924 +0000 UTC m=+989.603626855" Nov 01 22:40:24.205926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.205896 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:24.328645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.325806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:24.762503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:24.762474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:25.153504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:25.153420 4981 generic.go:356] "Generic (PLEG): container finished" podID="bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" containerID="2d904167066cb89d67a11d063d27b4382b421e814871652eed6d10f2610472e0" exitCode=0 Nov 01 22:40:25.154000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:25.153505 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" event={"ID":"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c","Type":"ContainerDied","Data":"2d904167066cb89d67a11d063d27b4382b421e814871652eed6d10f2610472e0"} Nov 01 22:40:25.589047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:25.589014 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:26.261983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.261890 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:26.372018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.371983 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:26.440545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.440517 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-kube-api-access-pf7t5\" (UniqueName: \"kubernetes.io/projected/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-kube-api-access-pf7t5\") pod \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " Nov 01 22:40:26.440847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.440727 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-xks-ca\") pod \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " Nov 01 22:40:26.441155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.441133 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-openshift-ca\") pod \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\" (UID: \"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c\") " Nov 01 22:40:26.441629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.441562 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-openshift-ca" pod "bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" (UID: "bc0d0707-bf21-41c9-9eff-ea900c0b8a9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:26.441836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.441808 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:26.443061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.443031 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-xks-ca" pod "bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" (UID: "bc0d0707-bf21-41c9-9eff-ea900c0b8a9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:26.443342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.443309 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-kube-api-access-pf7t5" pod "bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" (UID: "bc0d0707-bf21-41c9-9eff-ea900c0b8a9c"). InnerVolumeSpecName "kube-api-access-pf7t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:26.495097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.495066 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:26.543484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.543447 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf7t5\" (UniqueName: \"kubernetes.io/projected/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-kube-api-access-pf7t5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:26.543484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.543480 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:26.635863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.635831 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5"] Nov 01 22:40:26.636564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636545 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6db5039a-aa92-4889-b66e-187cbdaefdb7" containerName="revoke-keys" Nov 01 22:40:26.636564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636563 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db5039a-aa92-4889-b66e-187cbdaefdb7" containerName="revoke-keys" Nov 01 22:40:26.636564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636635 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c451365c-af96-46e5-8413-8e73372f7446" containerName="revoke-keys" Nov 01 22:40:26.636564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636642 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c451365c-af96-46e5-8413-8e73372f7446" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636661 4981 cpu_manager.go:414] "containerMap: removing container" podUID="337a224f-077a-4646-b0a3-ce2a7c1179a8" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636666 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337a224f-077a-4646-b0a3-ce2a7c1179a8" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636684 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636689 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636822 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="337a224f-077a-4646-b0a3-ce2a7c1179a8" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636834 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636846 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db5039a-aa92-4889-b66e-187cbdaefdb7" containerName="revoke-keys" Nov 01 22:40:26.636952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.636854 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c451365c-af96-46e5-8413-8e73372f7446" containerName="revoke-keys" Nov 01 22:40:26.640958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.640940 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.659849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.657943 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0f6226e-openshift-default-dockercfg-bd7tv\"" Nov 01 22:40:26.659849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.658990 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5"] Nov 01 22:40:26.691760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.691691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.745690 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb72q\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-kube-api-access-bb72q\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.745724 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-envoy\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.745792 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-token\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.745827 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-certs\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.745878 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-data\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.745969 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-socket\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.746015 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-podinfo\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.746038 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istiod-ca-cert\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.784326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.746153 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-credential-socket\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.828596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.818876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.854840 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-credential-socket\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855069 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb72q\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-kube-api-access-bb72q\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855104 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-envoy\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855139 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-token\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855182 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-certs\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855224 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-data\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855293 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-socket\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855371 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-podinfo\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.855408 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istiod-ca-cert\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.856331 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istiod-ca-cert\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.856657 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-credential-socket\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.859059 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-data\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.859532 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-certs\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.862072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.859765 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-socket\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.878601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.872515 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-token\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.887085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.882795 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-envoy\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.887085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.885982 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-podinfo\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.887085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.886615 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb72q\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-kube-api-access-bb72q\") pod \"e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:26.972991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:26.970296 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:27.179632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.179561 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" Nov 01 22:40:27.179632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.179563 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk" event={"ID":"bc0d0707-bf21-41c9-9eff-ea900c0b8a9c","Type":"ContainerDied","Data":"3bcbd9e305edd978d8005da93543d6e691a0aff38d7ba0bab7b600a99dc3d6e3"} Nov 01 22:40:27.179888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.179661 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bcbd9e305edd978d8005da93543d6e691a0aff38d7ba0bab7b600a99dc3d6e3" Nov 01 22:40:27.309632 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:27.309594 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0225bdd_0a3e_4013_9a2d_0623513fe8f3.slice/crio-bf4b847ba916ce2a21856399cf7e91519173a1641ada5a3e26565d0b4af16989 WatchSource:0}: Error finding container bf4b847ba916ce2a21856399cf7e91519173a1641ada5a3e26565d0b4af16989: Status 404 returned error can't find the container with id bf4b847ba916ce2a21856399cf7e91519173a1641ada5a3e26565d0b4af16989 Nov 01 22:40:27.319861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.312827 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:40:27.319861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.312915 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:40:27.323346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.321546 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:40:27.329203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.329180 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5"] Nov 01 22:40:27.707018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.706988 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:27.848605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:27.848541 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:28.118682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:28.118564 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:28.190774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:28.190740 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" event={"ID":"e0225bdd-0a3e-4013-9a2d-0623513fe8f3","Type":"ContainerStarted","Data":"1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac"} Nov 01 22:40:28.190954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:28.190820 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" event={"ID":"e0225bdd-0a3e-4013-9a2d-0623513fe8f3","Type":"ContainerStarted","Data":"bf4b847ba916ce2a21856399cf7e91519173a1641ada5a3e26565d0b4af16989"} Nov 01 22:40:28.222181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:28.222132 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" podStartSLOduration=2.222117436 podStartE2EDuration="2.222117436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:28.216749099 +0000 UTC m=+993.644627023" watchObservedRunningTime="2026-11-01 22:40:28.222117436 +0000 UTC m=+993.649995367" Nov 01 22:40:28.974799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:28.974766 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:28.981182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:28.981158 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:29.044738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.044689 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:29.121360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.121324 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv"] Nov 01 22:40:29.126342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.126321 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.133141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.133115 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-88896ec5\"" Nov 01 22:40:29.133671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.133649 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-88896ec5-dockercfg-b4wdm\"" Nov 01 22:40:29.136789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.136764 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv"] Nov 01 22:40:29.200704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.200663 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:29.206328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.206300 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:29.207439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.207417 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:40:29.354244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.349125 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmd6\" (UniqueName: \"kubernetes.io/projected/bd9cd489-b786-4bfb-8536-60b061ce13b2-kube-api-access-sdmd6\") pod \"payload-processing-e2e-ait-88896ec5-766999f795-pcrhv\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.354244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.349231 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9cd489-b786-4bfb-8536-60b061ce13b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-88896ec5-766999f795-pcrhv\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.470831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.455830 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmd6\" (UniqueName: \"kubernetes.io/projected/bd9cd489-b786-4bfb-8536-60b061ce13b2-kube-api-access-sdmd6\") pod \"payload-processing-e2e-ait-88896ec5-766999f795-pcrhv\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.470831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.455946 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9cd489-b786-4bfb-8536-60b061ce13b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-88896ec5-766999f795-pcrhv\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.470831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.463513 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9cd489-b786-4bfb-8536-60b061ce13b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-88896ec5-766999f795-pcrhv\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.475045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.474697 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmd6\" (UniqueName: \"kubernetes.io/projected/bd9cd489-b786-4bfb-8536-60b061ce13b2-kube-api-access-sdmd6\") pod \"payload-processing-e2e-ait-88896ec5-766999f795-pcrhv\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.747456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.747421 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:29.856694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:29.856661 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:30.023055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.022976 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:30.038431 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:30.038402 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9cd489_b786_4bfb_8536_60b061ce13b2.slice/crio-2028e4d5f39d362db2101921eb9c91eb6bb53a9633fd65f9bac4d6e4576d9c57 WatchSource:0}: Error finding container 2028e4d5f39d362db2101921eb9c91eb6bb53a9633fd65f9bac4d6e4576d9c57: Status 404 returned error can't find the container with id 2028e4d5f39d362db2101921eb9c91eb6bb53a9633fd65f9bac4d6e4576d9c57 Nov 01 22:40:30.040507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.040483 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv"] Nov 01 22:40:30.210617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.193349 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:30.380655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.376147 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" event={"ID":"bd9cd489-b786-4bfb-8536-60b061ce13b2","Type":"ContainerStarted","Data":"b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7"} Nov 01 22:40:30.380655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.376253 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" event={"ID":"bd9cd489-b786-4bfb-8536-60b061ce13b2","Type":"ContainerStarted","Data":"2028e4d5f39d362db2101921eb9c91eb6bb53a9633fd65f9bac4d6e4576d9c57"} Nov 01 22:40:30.380655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.376392 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:30.409057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.409007 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" podStartSLOduration=1.408993555 podStartE2EDuration="1.408993555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:30.405928617 +0000 UTC m=+995.833806550" watchObservedRunningTime="2026-11-01 22:40:30.408993555 +0000 UTC m=+995.836871485" Nov 01 22:40:30.976799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:30.976754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:31.192689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:31.192656 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:31.320887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:31.320785 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:31.442767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:31.442734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:31.553131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:31.553095 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:31.760597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:31.743727 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:32.893423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.893389 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:32.895369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.895346 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch"] Nov 01 22:40:32.901374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.901351 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:32.909312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.909292 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:40:32.909492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.909424 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:40:32.911990 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.911966 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch"] Nov 01 22:40:32.913293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.913272 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:32.913391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.913347 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:40:32.964366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.958684 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27gcg\" (UniqueName: \"kubernetes.io/projected/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-kube-api-access-27gcg\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:32.964366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.959045 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:32.964366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:32.959168 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.066017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.065976 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.066207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.066058 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.066258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.066230 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27gcg\" (UniqueName: \"kubernetes.io/projected/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-kube-api-access-27gcg\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.066832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.066812 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.068623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.068594 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.081511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.081488 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27gcg\" (UniqueName: \"kubernetes.io/projected/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-kube-api-access-27gcg\") pod \"maas-api-revoke-sub-8b9734fb-hb9ch\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.102605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.102567 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:33.127973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.127944 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr"] Nov 01 22:40:33.133105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.133085 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.307602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.257416 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:33.307602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.268020 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr"] Nov 01 22:40:33.307602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.304092 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:33.337246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.337208 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.421448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.421347 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.426033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.425995 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wg9w\" (UniqueName: \"kubernetes.io/projected/83b054b7-b214-4468-9787-09dad5ff2abe-kube-api-access-8wg9w\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.529819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.529236 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wg9w\" (UniqueName: \"kubernetes.io/projected/83b054b7-b214-4468-9787-09dad5ff2abe-kube-api-access-8wg9w\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.529819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.529325 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.529819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.529444 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.530595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.530411 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.532795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.532753 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.538174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.538150 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wg9w\" (UniqueName: \"kubernetes.io/projected/83b054b7-b214-4468-9787-09dad5ff2abe-kube-api-access-8wg9w\") pod \"maas-api-revoke-sub-113c024f-l7xjr\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:33.724439 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:33.724404 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d32a98_a3ef_4ae7_ac4e_c49d9fe9dac5.slice/crio-bb62687834d1d1cb84c0e6da5f7a06e415a789fc3504890244450597b2ef6a60 WatchSource:0}: Error finding container bb62687834d1d1cb84c0e6da5f7a06e415a789fc3504890244450597b2ef6a60: Status 404 returned error can't find the container with id bb62687834d1d1cb84c0e6da5f7a06e415a789fc3504890244450597b2ef6a60 Nov 01 22:40:33.727209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.727158 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch"] Nov 01 22:40:33.759958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:33.759781 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:34.171873 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:34.171841 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b054b7_b214_4468_9787_09dad5ff2abe.slice/crio-3df5d9bd9907dc72d2903711c87dee64529ca49b28d04cc18801a43b4a2e8465 WatchSource:0}: Error finding container 3df5d9bd9907dc72d2903711c87dee64529ca49b28d04cc18801a43b4a2e8465: Status 404 returned error can't find the container with id 3df5d9bd9907dc72d2903711c87dee64529ca49b28d04cc18801a43b4a2e8465 Nov 01 22:40:34.177382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.177341 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr"] Nov 01 22:40:34.451116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.451076 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" event={"ID":"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5","Type":"ContainerStarted","Data":"8eea1f306afd3627fc4c06c6f957a0fe363e2828b6a37b3b6f065fb71e5531ab"} Nov 01 22:40:34.451299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.451143 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" event={"ID":"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5","Type":"ContainerStarted","Data":"bb62687834d1d1cb84c0e6da5f7a06e415a789fc3504890244450597b2ef6a60"} Nov 01 22:40:34.452559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.452533 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" event={"ID":"83b054b7-b214-4468-9787-09dad5ff2abe","Type":"ContainerStarted","Data":"3df5d9bd9907dc72d2903711c87dee64529ca49b28d04cc18801a43b4a2e8465"} Nov 01 22:40:34.461044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.461022 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:34.492590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.492514 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" podStartSLOduration=2.492492203 podStartE2EDuration="2.492492203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:34.47132722 +0000 UTC m=+999.899205150" watchObservedRunningTime="2026-11-01 22:40:34.492492203 +0000 UTC m=+999.920370136" Nov 01 22:40:34.813566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.813542 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:34.951970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:34.951887 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:35.463012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:35.462922 4981 generic.go:356] "Generic (PLEG): container finished" podID="83b054b7-b214-4468-9787-09dad5ff2abe" containerID="b695e6dc275e64d50b9f0d99102ed791cfa0c1ef45387352360a2bef3e08527c" exitCode=0 Nov 01 22:40:35.463439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:35.463033 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" event={"ID":"83b054b7-b214-4468-9787-09dad5ff2abe","Type":"ContainerDied","Data":"b695e6dc275e64d50b9f0d99102ed791cfa0c1ef45387352360a2bef3e08527c"} Nov 01 22:40:35.464762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:35.464737 4981 generic.go:356] "Generic (PLEG): container finished" podID="99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" containerID="8eea1f306afd3627fc4c06c6f957a0fe363e2828b6a37b3b6f065fb71e5531ab" exitCode=0 Nov 01 22:40:35.464865 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:35.464778 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" event={"ID":"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5","Type":"ContainerDied","Data":"8eea1f306afd3627fc4c06c6f957a0fe363e2828b6a37b3b6f065fb71e5531ab"} Nov 01 22:40:35.637142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:35.637107 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:35.950813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:35.950768 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:36.001373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.001305 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54"] Nov 01 22:40:36.006497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.006475 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.022516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.022283 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54"] Nov 01 22:40:36.022704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.022683 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e61a2d4e\"" Nov 01 22:40:36.038506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.038478 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e61a2d4e-dockercfg-kphqs\"" Nov 01 22:40:36.056101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.056026 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:36.074850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.074823 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f8b0f5-9a48-413e-991d-944779679a9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.075020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.074987 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86g8w\" (UniqueName: \"kubernetes.io/projected/d5f8b0f5-9a48-413e-991d-944779679a9d-kube-api-access-86g8w\") pod \"payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.201753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.185945 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f8b0f5-9a48-413e-991d-944779679a9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.202298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.202265 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86g8w\" (UniqueName: \"kubernetes.io/projected/d5f8b0f5-9a48-413e-991d-944779679a9d-kube-api-access-86g8w\") pod \"payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.202628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.187528 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f8b0f5-9a48-413e-991d-944779679a9d-plugins-config-volume\") pod \"payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.224670 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.218443 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86g8w\" (UniqueName: \"kubernetes.io/projected/d5f8b0f5-9a48-413e-991d-944779679a9d-kube-api-access-86g8w\") pod \"payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.323187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.323139 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:36.644622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.644566 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54"] Nov 01 22:40:36.648613 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:36.648542 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f8b0f5_9a48_413e_991d_944779679a9d.slice/crio-692871c22d26aade24b2c1c77f6510755dca9945fbaf5a964a5a95a329003824 WatchSource:0}: Error finding container 692871c22d26aade24b2c1c77f6510755dca9945fbaf5a964a5a95a329003824: Status 404 returned error can't find the container with id 692871c22d26aade24b2c1c77f6510755dca9945fbaf5a964a5a95a329003824 Nov 01 22:40:36.724303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.724279 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:36.753119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.752680 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:36.765238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.765202 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:36.925071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925044 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-kube-api-access-27gcg\" (UniqueName: \"kubernetes.io/projected/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-kube-api-access-27gcg\") pod \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " Nov 01 22:40:36.925242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925149 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-openshift-ca\") pod \"83b054b7-b214-4468-9787-09dad5ff2abe\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " Nov 01 22:40:36.925242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925187 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-xks-ca\") pod \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " Nov 01 22:40:36.925340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925316 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-openshift-ca\") pod \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\" (UID: \"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5\") " Nov 01 22:40:36.925401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925386 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b054b7-b214-4468-9787-09dad5ff2abe-kube-api-access-8wg9w\" (UniqueName: \"kubernetes.io/projected/83b054b7-b214-4468-9787-09dad5ff2abe-kube-api-access-8wg9w\") pod \"83b054b7-b214-4468-9787-09dad5ff2abe\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " Nov 01 22:40:36.925524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925496 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-openshift-ca" pod "83b054b7-b214-4468-9787-09dad5ff2abe" (UID: "83b054b7-b214-4468-9787-09dad5ff2abe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:36.925705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925529 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-xks-ca\") pod \"83b054b7-b214-4468-9787-09dad5ff2abe\" (UID: \"83b054b7-b214-4468-9787-09dad5ff2abe\") " Nov 01 22:40:36.925766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.925706 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-openshift-ca" pod "99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" (UID: "99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:36.926762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.926700 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:36.926762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.926729 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:36.927654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.927630 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-kube-api-access-27gcg" pod "99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" (UID: "99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5"). InnerVolumeSpecName "kube-api-access-27gcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:36.927987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.927963 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-xks-ca" pod "99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" (UID: "99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:36.928067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.928036 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b054b7-b214-4468-9787-09dad5ff2abe-kube-api-access-8wg9w" pod "83b054b7-b214-4468-9787-09dad5ff2abe" (UID: "83b054b7-b214-4468-9787-09dad5ff2abe"). InnerVolumeSpecName "kube-api-access-8wg9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:36.928437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:36.928412 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-xks-ca" pod "83b054b7-b214-4468-9787-09dad5ff2abe" (UID: "83b054b7-b214-4468-9787-09dad5ff2abe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:37.028984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.028938 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wg9w\" (UniqueName: \"kubernetes.io/projected/83b054b7-b214-4468-9787-09dad5ff2abe-kube-api-access-8wg9w\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.028984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.028982 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b054b7-b214-4468-9787-09dad5ff2abe-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.029198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.029000 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27gcg\" (UniqueName: \"kubernetes.io/projected/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-kube-api-access-27gcg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.029198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.029009 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:37.494597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.494459 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:37.514239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.514207 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" Nov 01 22:40:37.514412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.514238 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr" event={"ID":"83b054b7-b214-4468-9787-09dad5ff2abe","Type":"ContainerDied","Data":"3df5d9bd9907dc72d2903711c87dee64529ca49b28d04cc18801a43b4a2e8465"} Nov 01 22:40:37.514412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.514287 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3df5d9bd9907dc72d2903711c87dee64529ca49b28d04cc18801a43b4a2e8465" Nov 01 22:40:37.515833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.515803 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" Nov 01 22:40:37.515960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.515802 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch" event={"ID":"99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5","Type":"ContainerDied","Data":"bb62687834d1d1cb84c0e6da5f7a06e415a789fc3504890244450597b2ef6a60"} Nov 01 22:40:37.515960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.515933 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb62687834d1d1cb84c0e6da5f7a06e415a789fc3504890244450597b2ef6a60" Nov 01 22:40:37.517883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.517856 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" event={"ID":"d5f8b0f5-9a48-413e-991d-944779679a9d","Type":"ContainerStarted","Data":"20b11c93d855f5c8b32aff087e4a7ea75356fce3f229657845ea87640ac6b046"} Nov 01 22:40:37.517989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.517912 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:37.517989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.517936 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" event={"ID":"d5f8b0f5-9a48-413e-991d-944779679a9d","Type":"ContainerStarted","Data":"692871c22d26aade24b2c1c77f6510755dca9945fbaf5a964a5a95a329003824"} Nov 01 22:40:37.544496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.544450 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" podStartSLOduration=2.544438146 podStartE2EDuration="2.544438146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:37.538376087 +0000 UTC m=+1002.966254022" watchObservedRunningTime="2026-11-01 22:40:37.544438146 +0000 UTC m=+1002.972316077" Nov 01 22:40:37.591850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:37.591823 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:38.675851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:38.675824 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:39.637402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:39.637366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:40.377804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:40.377767 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:40.489190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:40.489147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:40.586229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:40.586201 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:41.062549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:41.062515 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:41.158390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:41.158358 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:41.356165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:41.356080 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:41.395360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:41.395332 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:40:42.075029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:42.074989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:42.810634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:42.810557 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:43.102857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.102765 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:43.180607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.180553 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b"] Nov 01 22:40:43.181455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.181432 4981 cpu_manager.go:414] "containerMap: removing container" podUID="83b054b7-b214-4468-9787-09dad5ff2abe" containerName="revoke-keys" Nov 01 22:40:43.181455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.181455 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b054b7-b214-4468-9787-09dad5ff2abe" containerName="revoke-keys" Nov 01 22:40:43.181660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.181501 4981 cpu_manager.go:414] "containerMap: removing container" podUID="99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" containerName="revoke-keys" Nov 01 22:40:43.181660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.181510 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" containerName="revoke-keys" Nov 01 22:40:43.181760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.181726 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" containerName="revoke-keys" Nov 01 22:40:43.181760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.181757 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b054b7-b214-4468-9787-09dad5ff2abe" containerName="revoke-keys" Nov 01 22:40:43.186038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.186015 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.193495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.193473 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:43.193669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.193635 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:40:43.193764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.193743 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:40:43.194963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.194943 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:40:43.200766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.200588 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b"] Nov 01 22:40:43.229910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.229879 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:43.282838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.282805 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkkj\" (UniqueName: \"kubernetes.io/projected/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-kube-api-access-xfkkj\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.283004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.282846 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.283095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.283074 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.363790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.363708 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:43.384628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.384563 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.384808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.384790 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkkj\" (UniqueName: \"kubernetes.io/projected/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-kube-api-access-xfkkj\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.384870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.384822 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.385229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.385209 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.387172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.387150 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.395297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.395276 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkkj\" (UniqueName: \"kubernetes.io/projected/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-kube-api-access-xfkkj\") pod \"maas-api-revoke-sub-00ddad3f-gbw9b\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.507329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.507299 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:43.807809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.807782 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b"] Nov 01 22:40:43.810184 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:43.810153 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9229dc2_9126_4d66_a93a_a433e6a3dcd7.slice/crio-61770cd3222ebf2acb9b89f88cb0d39707dacfeaaff7cfe9579dee812b9edb48 WatchSource:0}: Error finding container 61770cd3222ebf2acb9b89f88cb0d39707dacfeaaff7cfe9579dee812b9edb48: Status 404 returned error can't find the container with id 61770cd3222ebf2acb9b89f88cb0d39707dacfeaaff7cfe9579dee812b9edb48 Nov 01 22:40:43.824688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.824659 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:43.936700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:43.936669 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:44.297002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:44.296972 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:44.400164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:44.400134 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:44.659686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:44.658315 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" event={"ID":"c9229dc2-9126-4d66-a93a-a433e6a3dcd7","Type":"ContainerStarted","Data":"211a28546bd36d8752c5fa2e4a0e7406dce28e0075166f29e698c670c4f009c9"} Nov 01 22:40:44.659686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:44.658384 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" event={"ID":"c9229dc2-9126-4d66-a93a-a433e6a3dcd7","Type":"ContainerStarted","Data":"61770cd3222ebf2acb9b89f88cb0d39707dacfeaaff7cfe9579dee812b9edb48"} Nov 01 22:40:44.680422 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:40:44.676793 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:40:45.295212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:45.295183 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:45.611064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:45.610993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:45.670925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:45.670893 4981 generic.go:356] "Generic (PLEG): container finished" podID="c9229dc2-9126-4d66-a93a-a433e6a3dcd7" containerID="211a28546bd36d8752c5fa2e4a0e7406dce28e0075166f29e698c670c4f009c9" exitCode=0 Nov 01 22:40:45.671100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:45.670963 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" event={"ID":"c9229dc2-9126-4d66-a93a-a433e6a3dcd7","Type":"ContainerDied","Data":"211a28546bd36d8752c5fa2e4a0e7406dce28e0075166f29e698c670c4f009c9"} Nov 01 22:40:45.728937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:45.728905 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:45.970747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:45.970715 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:46.262859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.262766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:46.289642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.289023 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6"] Nov 01 22:40:46.296837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.296809 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.300067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.300036 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6"] Nov 01 22:40:46.413161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.402301 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.413161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.402446 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.413161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.402758 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfp9p\" (UniqueName: \"kubernetes.io/projected/c6f69691-5323-400a-8021-83623c2b3c8b-kube-api-access-nfp9p\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.535651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.534948 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.535651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.535084 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.535651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.535308 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfp9p\" (UniqueName: \"kubernetes.io/projected/c6f69691-5323-400a-8021-83623c2b3c8b-kube-api-access-nfp9p\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.538856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.538828 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.568177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.568136 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.636387 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.636347 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfp9p\" (UniqueName: \"kubernetes.io/projected/c6f69691-5323-400a-8021-83623c2b3c8b-kube-api-access-nfp9p\") pod \"maas-api-revoke-sub-4fdc0c68-lpqh6\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:46.946601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:46.930105 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:47.832617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:47.826544 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:47.900598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:47.875124 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:40:47.933761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:47.933713 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.023344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.023239 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.090604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.036134 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:40:48.172700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.059455 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:40:48.172700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.059495 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:40:48.172700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.059518 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:40:48.174979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.059538 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:40:48.216599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.059840 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:40:48.216599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.090975 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:40:48.216599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.091031 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.216599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.154368 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:40:48.216599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.154909 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.666884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.666781 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:40:48.739665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.712917 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.852727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.852668 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:48.890737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.890674 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:48.919885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.919799 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:48.967155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.943823 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:40:49.012149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:48.997555 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-xks-ca\") pod \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " Nov 01 22:40:49.012149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.010486 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-openshift-ca\") pod \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " Nov 01 22:40:49.012149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.010592 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-kube-api-access-xfkkj\" (UniqueName: \"kubernetes.io/projected/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-kube-api-access-xfkkj\") pod \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\" (UID: \"c9229dc2-9126-4d66-a93a-a433e6a3dcd7\") " Nov 01 22:40:49.012949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.012632 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-openshift-ca" pod "c9229dc2-9126-4d66-a93a-a433e6a3dcd7" (UID: "c9229dc2-9126-4d66-a93a-a433e6a3dcd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:49.042606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.024785 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.087602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.082527 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:40:49.088299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.088272 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:40:49.093608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.088483 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:40:49.204083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.203021 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:40:49.269663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.237701 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.269663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.239913 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-xks-ca" pod "c9229dc2-9126-4d66-a93a-a433e6a3dcd7" (UID: "c9229dc2-9126-4d66-a93a-a433e6a3dcd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:49.269663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.240073 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-kube-api-access-xfkkj" pod "c9229dc2-9126-4d66-a93a-a433e6a3dcd7" (UID: "c9229dc2-9126-4d66-a93a-a433e6a3dcd7"). InnerVolumeSpecName "kube-api-access-xfkkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:49.280741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.280684 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.372495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.286919 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:40:49.372495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.287169 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:40:49.598614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.572471 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 01 22:40:49.696060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.683709 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:40:49.696060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.684840 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:40:49.696060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.688676 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:40:49.696060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.688943 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:40:49.741181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.720901 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:40:49.741181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.737686 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:49.741562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.741520 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:40:49.777642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.744251 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:40:49.939043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.939011 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.017959 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018045 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018103 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018160 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018211 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018260 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018310 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018361 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.018414 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.896709 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:49.792699 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f69691_5323_400a_8021_83623c2b3c8b.slice/crio-0089e7a1532d74193b74ead5a4a7a46215b2d67e1b72a3e35cddb75cfc029171 WatchSource:0}: Error finding container 0089e7a1532d74193b74ead5a4a7a46215b2d67e1b72a3e35cddb75cfc029171: Status 404 returned error can't find the container with id 0089e7a1532d74193b74ead5a4a7a46215b2d67e1b72a3e35cddb75cfc029171 Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.974958 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:40:50.021672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:49.989883 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.175410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.131645 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.303769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.237857 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:50.303769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.238034 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.303769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.241682 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.380774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.380424 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:50.380774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.380514 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.412231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.412194 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:50.459348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.456487 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.459348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.456604 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:40:50.459348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.456645 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:40:50.482600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.476162 4981 patch_prober.go:29] interesting pod/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:40:50.482600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.476240 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.582663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.540554 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:50.658874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.658826 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:50.803605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.714306 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:40:51.059404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.059369 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.112478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.886430 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:40:51.114163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.886465 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:40:51.116662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.886510 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.120903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.931043 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.568375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.934710 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.576716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.989897 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.631148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.064642 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:51.631394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.631371 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfkkj\" (UniqueName: \"kubernetes.io/projected/c9229dc2-9126-4d66-a93a-a433e6a3dcd7-kube-api-access-xfkkj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:51.643996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.059329 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.662405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:50.728128 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.662405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.169695 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:51.662405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.270693 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:40:51.662405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.271255 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:51.662405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.302743 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.684782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.342257 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" Nov 01 22:40:51.708245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.072740 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-cxr2r" podUID="1406f61b-2ddc-402f-8631-3b00159964b0" containerName="server" probeResult="failure" output="Get \"https://10.134.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:51.807259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.807204 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:40:51.824242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.820450 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:40:51.824242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:51.822366 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.117757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.117704 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.130973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.130932 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:40:52.134810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.134773 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:40:52.230886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.213175 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:40:52.230886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.214455 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:40:52.453598 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:40:52.422602 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9229dc2_9126_4d66_a93a_a433e6a3dcd7.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:40:52.589865 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.586971 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.650095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.650056 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.658733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.658692 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.660149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.660119 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:40:52.781704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.676669 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:40:52.809913 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:40:52.699512 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9229dc2_9126_4d66_a93a_a433e6a3dcd7.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:40:52.822483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.738690 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.824410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.746396 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:40:52.830478 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:40:52.770850 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.884s" Nov 01 22:40:52.830478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.827990 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.863788 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b" event={"ID":"c9229dc2-9126-4d66-a93a-a433e6a3dcd7","Type":"ContainerDied","Data":"61770cd3222ebf2acb9b89f88cb0d39707dacfeaaff7cfe9579dee812b9edb48"} Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.863839 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61770cd3222ebf2acb9b89f88cb0d39707dacfeaaff7cfe9579dee812b9edb48" Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.863858 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6"] Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.863917 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" event={"ID":"c6f69691-5323-400a-8021-83623c2b3c8b","Type":"ContainerStarted","Data":"0089e7a1532d74193b74ead5a4a7a46215b2d67e1b72a3e35cddb75cfc029171"} Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864016 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864074 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864123 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:40:52.864182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864175 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864235 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864285 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864350 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864396 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864443 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864492 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864541 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864613 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:40:52.864727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.864665 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:40:52.875487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.875454 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:40:52.879636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.879610 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:40:52.961491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:52.961454 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:53.102563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:53.102521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:53.687424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:53.687332 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:53.854952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:53.854911 4981 generic.go:356] "Generic (PLEG): container finished" podID="c6f69691-5323-400a-8021-83623c2b3c8b" containerID="38f8dd33671d33be8007773adb87966853ae889c833aed4a4cf876d6579332ff" exitCode=0 Nov 01 22:40:53.855135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:53.854996 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" event={"ID":"c6f69691-5323-400a-8021-83623c2b3c8b","Type":"ContainerDied","Data":"38f8dd33671d33be8007773adb87966853ae889c833aed4a4cf876d6579332ff"} Nov 01 22:40:54.840543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:54.840506 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:55.088078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.088048 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:55.091874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.091819 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:55.223607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.223538 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:55.232156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.232132 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-xks-ca\") pod \"c6f69691-5323-400a-8021-83623c2b3c8b\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " Nov 01 22:40:55.232249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.232180 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-openshift-ca\") pod \"c6f69691-5323-400a-8021-83623c2b3c8b\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " Nov 01 22:40:55.232329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.232318 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f69691-5323-400a-8021-83623c2b3c8b-kube-api-access-nfp9p\" (UniqueName: \"kubernetes.io/projected/c6f69691-5323-400a-8021-83623c2b3c8b-kube-api-access-nfp9p\") pod \"c6f69691-5323-400a-8021-83623c2b3c8b\" (UID: \"c6f69691-5323-400a-8021-83623c2b3c8b\") " Nov 01 22:40:55.232605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.232561 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-openshift-ca" pod "c6f69691-5323-400a-8021-83623c2b3c8b" (UID: "c6f69691-5323-400a-8021-83623c2b3c8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:40:55.232911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.232892 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:55.234476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.234455 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-xks-ca" pod "c6f69691-5323-400a-8021-83623c2b3c8b" (UID: "c6f69691-5323-400a-8021-83623c2b3c8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:40:55.234550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.234518 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f69691-5323-400a-8021-83623c2b3c8b-kube-api-access-nfp9p" pod "c6f69691-5323-400a-8021-83623c2b3c8b" (UID: "c6f69691-5323-400a-8021-83623c2b3c8b"). InnerVolumeSpecName "kube-api-access-nfp9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:40:55.332492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.332463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:55.334435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.334410 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfp9p\" (UniqueName: \"kubernetes.io/projected/c6f69691-5323-400a-8021-83623c2b3c8b-kube-api-access-nfp9p\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:55.334435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.334436 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f69691-5323-400a-8021-83623c2b3c8b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:40:55.877551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.877518 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" Nov 01 22:40:55.877551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.877545 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6" event={"ID":"c6f69691-5323-400a-8021-83623c2b3c8b","Type":"ContainerDied","Data":"0089e7a1532d74193b74ead5a4a7a46215b2d67e1b72a3e35cddb75cfc029171"} Nov 01 22:40:55.878024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.877605 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0089e7a1532d74193b74ead5a4a7a46215b2d67e1b72a3e35cddb75cfc029171" Nov 01 22:40:55.956879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:55.956839 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:56.339327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.339277 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57"] Nov 01 22:40:56.340961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.340929 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c9229dc2-9126-4d66-a93a-a433e6a3dcd7" containerName="revoke-keys" Nov 01 22:40:56.340961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.340955 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9229dc2-9126-4d66-a93a-a433e6a3dcd7" containerName="revoke-keys" Nov 01 22:40:56.341362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.341086 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c6f69691-5323-400a-8021-83623c2b3c8b" containerName="revoke-keys" Nov 01 22:40:56.341362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.341099 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f69691-5323-400a-8021-83623c2b3c8b" containerName="revoke-keys" Nov 01 22:40:56.341362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.341321 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9229dc2-9126-4d66-a93a-a433e6a3dcd7" containerName="revoke-keys" Nov 01 22:40:56.341362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.341348 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f69691-5323-400a-8021-83623c2b3c8b" containerName="revoke-keys" Nov 01 22:40:56.346942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.346921 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.350097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.350054 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57"] Nov 01 22:40:56.423399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.423368 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 22:40:56.423544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.423438 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 22:40:56.423544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.423454 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 22:40:56.423544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.423479 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xhx7b\"" Nov 01 22:40:56.455242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.455203 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twflh\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-kube-api-access-twflh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.455410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.455275 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.455465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.455424 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.455465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.455458 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.455562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.455492 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.455661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.455632 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.455809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.455792 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461c591-54fb-4cc8-a218-3e5bc6583857-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.464212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.464186 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:56.557954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.557914 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twflh\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-kube-api-access-twflh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.558124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.557972 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.558171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.558125 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.558213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.558179 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.558251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.558241 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.558303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.558286 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.558484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.558466 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461c591-54fb-4cc8-a218-3e5bc6583857-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.559051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.559017 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.560785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.560757 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.560785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.560779 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.561149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.561125 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461c591-54fb-4cc8-a218-3e5bc6583857-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.561224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.561131 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.566813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.566790 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.567375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.567349 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twflh\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-kube-api-access-twflh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b8x57\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.569057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.569023 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:56.666114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.666030 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:40:56.873660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.873625 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:56.967149 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:56.967114 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1461c591_54fb_4cc8_a218_3e5bc6583857.slice/crio-5e652217b2bcdad7971610ae0f6d1c39cf42cef1836ec68753eee057cbefc9b7 WatchSource:0}: Error finding container 5e652217b2bcdad7971610ae0f6d1c39cf42cef1836ec68753eee057cbefc9b7: Status 404 returned error can't find the container with id 5e652217b2bcdad7971610ae0f6d1c39cf42cef1836ec68753eee057cbefc9b7 Nov 01 22:40:56.968911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:56.968879 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57"] Nov 01 22:40:57.916883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:57.916837 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" event={"ID":"1461c591-54fb-4cc8-a218-3e5bc6583857","Type":"ContainerStarted","Data":"5e652217b2bcdad7971610ae0f6d1c39cf42cef1836ec68753eee057cbefc9b7"} Nov 01 22:40:58.312193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.312148 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:58.631027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.630665 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5"] Nov 01 22:40:58.637644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.636912 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.644993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.644943 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:40:58.651347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.650939 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5"] Nov 01 22:40:58.658876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.658842 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:40:58.658992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.658919 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:40:58.659615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.659570 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:40:58.703878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.700994 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.703878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.701062 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.703878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.701173 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qwv9\" (UniqueName: \"kubernetes.io/projected/b5bad23d-8d92-40ff-a62e-31bcbe226913-kube-api-access-5qwv9\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.803138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.803098 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.803335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.803157 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.803335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.803305 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwv9\" (UniqueName: \"kubernetes.io/projected/b5bad23d-8d92-40ff-a62e-31bcbe226913-kube-api-access-5qwv9\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.803847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.803816 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.805977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.805955 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.815353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.815324 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qwv9\" (UniqueName: \"kubernetes.io/projected/b5bad23d-8d92-40ff-a62e-31bcbe226913-kube-api-access-5qwv9\") pod \"maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:58.965885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:58.965847 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:40:59.264404 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:40:59.264370 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bad23d_8d92_40ff_a62e_31bcbe226913.slice/crio-5fb2fdf91396b2aeda4fe76c4fbadf969e882263d90bc3109a271f5d28630c95 WatchSource:0}: Error finding container 5fb2fdf91396b2aeda4fe76c4fbadf969e882263d90bc3109a271f5d28630c95: Status 404 returned error can't find the container with id 5fb2fdf91396b2aeda4fe76c4fbadf969e882263d90bc3109a271f5d28630c95 Nov 01 22:40:59.268399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:59.268306 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5"] Nov 01 22:40:59.370292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:59.370248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:59.681880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:59.681842 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:59.818538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:59.818506 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:40:59.944176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:59.944129 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" event={"ID":"b5bad23d-8d92-40ff-a62e-31bcbe226913","Type":"ContainerStarted","Data":"12d3c4a2db9e8e490effae70be6490c2581ee7fcae488b9296f0f5441dd24d30"} Nov 01 22:40:59.944377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:59.944203 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" event={"ID":"b5bad23d-8d92-40ff-a62e-31bcbe226913","Type":"ContainerStarted","Data":"5fb2fdf91396b2aeda4fe76c4fbadf969e882263d90bc3109a271f5d28630c95"} Nov 01 22:40:59.972105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:40:59.972025 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" podStartSLOduration=1.972004606 podStartE2EDuration="1.972004606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:59.965105276 +0000 UTC m=+1025.392983280" watchObservedRunningTime="2026-11-01 22:40:59.972004606 +0000 UTC m=+1025.399882539" Nov 01 22:41:00.203098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:00.203066 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:41:00.203173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:00.203133 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:41:00.311810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:00.311776 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:00.957852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:00.957811 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" event={"ID":"1461c591-54fb-4cc8-a218-3e5bc6583857","Type":"ContainerStarted","Data":"b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7"} Nov 01 22:41:00.958356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:00.957913 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:41:00.960042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:00.960015 4981 generic.go:356] "Generic (PLEG): container finished" podID="b5bad23d-8d92-40ff-a62e-31bcbe226913" containerID="12d3c4a2db9e8e490effae70be6490c2581ee7fcae488b9296f0f5441dd24d30" exitCode=0 Nov 01 22:41:00.960196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:00.960111 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" event={"ID":"b5bad23d-8d92-40ff-a62e-31bcbe226913","Type":"ContainerDied","Data":"12d3c4a2db9e8e490effae70be6490c2581ee7fcae488b9296f0f5441dd24d30"} Nov 01 22:41:01.003502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.003440 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" podStartSLOduration=1.772254199 podStartE2EDuration="5.003422484s" totalImagesPullingTime="3.231168285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:00.996152049 +0000 UTC m=+1026.424029981" watchObservedRunningTime="2026-11-01 22:41:01.003422484 +0000 UTC m=+1026.431300418" Nov 01 22:41:01.122077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.122045 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq"] Nov 01 22:41:01.131070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.131042 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.154606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.153526 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.154606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.153723 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-kube-api-access-9j97r\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.154606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.153895 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.160897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.160871 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq"] Nov 01 22:41:01.258951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.258839 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.259126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.259010 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-kube-api-access-9j97r\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.259200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.259155 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.259752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.259721 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.262629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.262314 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.277188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.277160 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-kube-api-access-9j97r\") pod \"maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.395268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.395229 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:01.482891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.482856 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:01.781962 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:41:01.781927 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fef3bd5_0401_4b2c_bb69_8ef2acce50a1.slice/crio-97fd41ef8114fe5c4f007ea11d30b125fcccdf58239b21cb80e9a8b665603b7b WatchSource:0}: Error finding container 97fd41ef8114fe5c4f007ea11d30b125fcccdf58239b21cb80e9a8b665603b7b: Status 404 returned error can't find the container with id 97fd41ef8114fe5c4f007ea11d30b125fcccdf58239b21cb80e9a8b665603b7b Nov 01 22:41:01.785629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.785603 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq"] Nov 01 22:41:01.971627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.971565 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" event={"ID":"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1","Type":"ContainerStarted","Data":"97fd41ef8114fe5c4f007ea11d30b125fcccdf58239b21cb80e9a8b665603b7b"} Nov 01 22:41:01.973891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:01.973868 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 22:41:02.159456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.159427 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:41:02.341610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.333968 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-openshift-ca\") pod \"b5bad23d-8d92-40ff-a62e-31bcbe226913\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " Nov 01 22:41:02.341610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.334221 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bad23d-8d92-40ff-a62e-31bcbe226913-kube-api-access-5qwv9\" (UniqueName: \"kubernetes.io/projected/b5bad23d-8d92-40ff-a62e-31bcbe226913-kube-api-access-5qwv9\") pod \"b5bad23d-8d92-40ff-a62e-31bcbe226913\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " Nov 01 22:41:02.341610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.334374 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-xks-ca\") pod \"b5bad23d-8d92-40ff-a62e-31bcbe226913\" (UID: \"b5bad23d-8d92-40ff-a62e-31bcbe226913\") " Nov 01 22:41:02.349915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.349816 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-openshift-ca" pod "b5bad23d-8d92-40ff-a62e-31bcbe226913" (UID: "b5bad23d-8d92-40ff-a62e-31bcbe226913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:02.383330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.375495 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:02.402889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.402519 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-xks-ca" pod "b5bad23d-8d92-40ff-a62e-31bcbe226913" (UID: "b5bad23d-8d92-40ff-a62e-31bcbe226913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:02.433703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.424955 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bad23d-8d92-40ff-a62e-31bcbe226913-kube-api-access-5qwv9" pod "b5bad23d-8d92-40ff-a62e-31bcbe226913" (UID: "b5bad23d-8d92-40ff-a62e-31bcbe226913"). InnerVolumeSpecName "kube-api-access-5qwv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:02.477418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.477381 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qwv9\" (UniqueName: \"kubernetes.io/projected/b5bad23d-8d92-40ff-a62e-31bcbe226913-kube-api-access-5qwv9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:02.477418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.477416 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5bad23d-8d92-40ff-a62e-31bcbe226913-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:02.679992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.679961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:02.982927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.982846 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" Nov 01 22:41:02.982927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.982872 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5" event={"ID":"b5bad23d-8d92-40ff-a62e-31bcbe226913","Type":"ContainerDied","Data":"5fb2fdf91396b2aeda4fe76c4fbadf969e882263d90bc3109a271f5d28630c95"} Nov 01 22:41:02.983383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.982934 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fb2fdf91396b2aeda4fe76c4fbadf969e882263d90bc3109a271f5d28630c95" Nov 01 22:41:02.984701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.984662 4981 generic.go:356] "Generic (PLEG): container finished" podID="7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" containerID="8ae8d1690ccc4d64c820c64031ff9c9d09672a08775e9a26d5763996ae65cd3a" exitCode=0 Nov 01 22:41:02.984820 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:02.984724 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" event={"ID":"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1","Type":"ContainerDied","Data":"8ae8d1690ccc4d64c820c64031ff9c9d09672a08775e9a26d5763996ae65cd3a"} Nov 01 22:41:03.600522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:03.600484 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:03.810961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:03.810931 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:04.164051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.164016 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:04.208978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.208938 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-xks-ca\") pod \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " Nov 01 22:41:04.209170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.209128 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-kube-api-access-9j97r\") pod \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " Nov 01 22:41:04.209242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.209190 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-openshift-ca\") pod \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\" (UID: \"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1\") " Nov 01 22:41:04.209531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.209502 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-openshift-ca" pod "7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" (UID: "7fef3bd5-0401-4b2c-bb69-8ef2acce50a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:04.210150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.210124 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:04.211159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.211135 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-xks-ca" pod "7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" (UID: "7fef3bd5-0401-4b2c-bb69-8ef2acce50a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:04.211245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.211211 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-kube-api-access-9j97r" pod "7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" (UID: "7fef3bd5-0401-4b2c-bb69-8ef2acce50a1"). InnerVolumeSpecName "kube-api-access-9j97r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:04.311552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.311470 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:04.311552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.311503 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j97r\" (UniqueName: \"kubernetes.io/projected/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1-kube-api-access-9j97r\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:04.556197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.556161 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:04.865507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.865465 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v"] Nov 01 22:41:04.866441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.866412 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b5bad23d-8d92-40ff-a62e-31bcbe226913" containerName="revoke-keys" Nov 01 22:41:04.866441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.866437 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bad23d-8d92-40ff-a62e-31bcbe226913" containerName="revoke-keys" Nov 01 22:41:04.866706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.866504 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" containerName="revoke-keys" Nov 01 22:41:04.866706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.866513 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" containerName="revoke-keys" Nov 01 22:41:04.866814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.866756 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bad23d-8d92-40ff-a62e-31bcbe226913" containerName="revoke-keys" Nov 01 22:41:04.866814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.866784 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" containerName="revoke-keys" Nov 01 22:41:04.871064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.871045 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:04.882715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.882690 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v"] Nov 01 22:41:04.928471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.928438 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:04.928729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.928709 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d957r\" (UniqueName: \"kubernetes.io/projected/b9914a5f-608f-4738-a1ac-78b0930451bf-kube-api-access-d957r\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:04.928949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:04.928926 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.004182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.004101 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" Nov 01 22:41:05.004182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.004133 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq" event={"ID":"7fef3bd5-0401-4b2c-bb69-8ef2acce50a1","Type":"ContainerDied","Data":"97fd41ef8114fe5c4f007ea11d30b125fcccdf58239b21cb80e9a8b665603b7b"} Nov 01 22:41:05.004368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.004190 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97fd41ef8114fe5c4f007ea11d30b125fcccdf58239b21cb80e9a8b665603b7b" Nov 01 22:41:05.030521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.030490 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.030718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.030533 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.030791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.030772 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d957r\" (UniqueName: \"kubernetes.io/projected/b9914a5f-608f-4738-a1ac-78b0930451bf-kube-api-access-d957r\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.031239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.031217 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.033064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.033035 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.039700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.039679 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d957r\" (UniqueName: \"kubernetes.io/projected/b9914a5f-608f-4738-a1ac-78b0930451bf-kube-api-access-d957r\") pod \"maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.189158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.189123 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:05.479152 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:41:05.479123 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9914a5f_608f_4738_a1ac_78b0930451bf.slice/crio-245c3c6bd10d87c137b063410e1ffc2e378b702e9e0a632f9ca7f18664869433 WatchSource:0}: Error finding container 245c3c6bd10d87c137b063410e1ffc2e378b702e9e0a632f9ca7f18664869433: Status 404 returned error can't find the container with id 245c3c6bd10d87c137b063410e1ffc2e378b702e9e0a632f9ca7f18664869433 Nov 01 22:41:05.482781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.482757 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v"] Nov 01 22:41:05.542861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:05.542833 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:06.015037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:06.015001 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" event={"ID":"b9914a5f-608f-4738-a1ac-78b0930451bf","Type":"ContainerStarted","Data":"06b05a5276823ecae11352f5a4fb37a87fc9db8bbd4992b9500836f4ef878ab5"} Nov 01 22:41:06.015216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:06.015062 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" event={"ID":"b9914a5f-608f-4738-a1ac-78b0930451bf","Type":"ContainerStarted","Data":"245c3c6bd10d87c137b063410e1ffc2e378b702e9e0a632f9ca7f18664869433"} Nov 01 22:41:06.044011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:06.043959 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" podStartSLOduration=2.043946246 podStartE2EDuration="2.043946246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:06.038347547 +0000 UTC m=+1031.466225470" watchObservedRunningTime="2026-11-01 22:41:06.043946246 +0000 UTC m=+1031.471824176" Nov 01 22:41:06.463355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:06.463326 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:07.028374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:07.028342 4981 generic.go:356] "Generic (PLEG): container finished" podID="b9914a5f-608f-4738-a1ac-78b0930451bf" containerID="06b05a5276823ecae11352f5a4fb37a87fc9db8bbd4992b9500836f4ef878ab5" exitCode=0 Nov 01 22:41:07.028561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:07.028422 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" event={"ID":"b9914a5f-608f-4738-a1ac-78b0930451bf","Type":"ContainerDied","Data":"06b05a5276823ecae11352f5a4fb37a87fc9db8bbd4992b9500836f4ef878ab5"} Nov 01 22:41:07.311012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:07.310931 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:07.985201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:07.985166 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:08.277611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.277551 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:08.385339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.385306 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-xks-ca\") pod \"b9914a5f-608f-4738-a1ac-78b0930451bf\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " Nov 01 22:41:08.385517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.385508 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-openshift-ca\") pod \"b9914a5f-608f-4738-a1ac-78b0930451bf\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " Nov 01 22:41:08.385709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.385693 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9914a5f-608f-4738-a1ac-78b0930451bf-kube-api-access-d957r\" (UniqueName: \"kubernetes.io/projected/b9914a5f-608f-4738-a1ac-78b0930451bf-kube-api-access-d957r\") pod \"b9914a5f-608f-4738-a1ac-78b0930451bf\" (UID: \"b9914a5f-608f-4738-a1ac-78b0930451bf\") " Nov 01 22:41:08.385936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.385907 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-openshift-ca" pod "b9914a5f-608f-4738-a1ac-78b0930451bf" (UID: "b9914a5f-608f-4738-a1ac-78b0930451bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:08.386350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.386331 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:08.387452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.387433 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-xks-ca" pod "b9914a5f-608f-4738-a1ac-78b0930451bf" (UID: "b9914a5f-608f-4738-a1ac-78b0930451bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:08.387818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.387799 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9914a5f-608f-4738-a1ac-78b0930451bf-kube-api-access-d957r" pod "b9914a5f-608f-4738-a1ac-78b0930451bf" (UID: "b9914a5f-608f-4738-a1ac-78b0930451bf"). InnerVolumeSpecName "kube-api-access-d957r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:08.488116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.488088 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d957r\" (UniqueName: \"kubernetes.io/projected/b9914a5f-608f-4738-a1ac-78b0930451bf-kube-api-access-d957r\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:08.488116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.488113 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9914a5f-608f-4738-a1ac-78b0930451bf-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:08.746729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.746697 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv"] Nov 01 22:41:08.880487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.880445 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:08.984979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:08.984945 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:09.190941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:09.190615 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" event={"ID":"b9914a5f-608f-4738-a1ac-78b0930451bf","Type":"ContainerDied","Data":"245c3c6bd10d87c137b063410e1ffc2e378b702e9e0a632f9ca7f18664869433"} Nov 01 22:41:09.190941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:09.190686 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245c3c6bd10d87c137b063410e1ffc2e378b702e9e0a632f9ca7f18664869433" Nov 01 22:41:09.190941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:09.190762 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v" Nov 01 22:41:09.667912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:09.667812 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:10.586287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:10.586246 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:11.361246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:11.361207 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:11.461354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:11.461323 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:12.144273 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:12.144234 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:12.879635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:12.879602 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:13.043186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:13.043151 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:13.156566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:13.156477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:13.522071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:13.522036 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:13.747930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:13.747884 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" podUID="bd9cd489-b786-4bfb-8536-60b061ce13b2" containerName="payload-processing" containerID="cri-o://b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7" gracePeriod=25 Nov 01 22:41:14.077069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.077036 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:41:14.102465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.102379 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:14.204142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.204105 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9cd489-b786-4bfb-8536-60b061ce13b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9cd489-b786-4bfb-8536-60b061ce13b2-plugins-config-volume\") pod \"bd9cd489-b786-4bfb-8536-60b061ce13b2\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " Nov 01 22:41:14.204515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.204237 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9cd489-b786-4bfb-8536-60b061ce13b2-kube-api-access-sdmd6\" (UniqueName: \"kubernetes.io/projected/bd9cd489-b786-4bfb-8536-60b061ce13b2-kube-api-access-sdmd6\") pod \"bd9cd489-b786-4bfb-8536-60b061ce13b2\" (UID: \"bd9cd489-b786-4bfb-8536-60b061ce13b2\") " Nov 01 22:41:14.204568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.204540 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9cd489-b786-4bfb-8536-60b061ce13b2-plugins-config-volume" pod "bd9cd489-b786-4bfb-8536-60b061ce13b2" (UID: "bd9cd489-b786-4bfb-8536-60b061ce13b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:14.204960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.204941 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd9cd489-b786-4bfb-8536-60b061ce13b2-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.206459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.206429 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9cd489-b786-4bfb-8536-60b061ce13b2-kube-api-access-sdmd6" pod "bd9cd489-b786-4bfb-8536-60b061ce13b2" (UID: "bd9cd489-b786-4bfb-8536-60b061ce13b2"). InnerVolumeSpecName "kube-api-access-sdmd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:14.248651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.248615 4981 generic.go:356] "Generic (PLEG): container finished" podID="bd9cd489-b786-4bfb-8536-60b061ce13b2" containerID="b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7" exitCode=0 Nov 01 22:41:14.248814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.248670 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" Nov 01 22:41:14.248814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.248670 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" event={"ID":"bd9cd489-b786-4bfb-8536-60b061ce13b2","Type":"ContainerDied","Data":"b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7"} Nov 01 22:41:14.248898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.248831 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv" event={"ID":"bd9cd489-b786-4bfb-8536-60b061ce13b2","Type":"ContainerDied","Data":"2028e4d5f39d362db2101921eb9c91eb6bb53a9633fd65f9bac4d6e4576d9c57"} Nov 01 22:41:14.248898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.248860 4981 scope.go:122] "RemoveContainer" containerID="b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7" Nov 01 22:41:14.282606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.272857 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:14.411686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.389195 4981 scope.go:122] "RemoveContainer" containerID="b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7" Nov 01 22:41:14.411686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.392989 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmd6\" (UniqueName: \"kubernetes.io/projected/bd9cd489-b786-4bfb-8536-60b061ce13b2-kube-api-access-sdmd6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:14.411686 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:41:14.392998 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7\": container with ID starting with b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7 not found: ID does not exist" containerID="b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7" Nov 01 22:41:14.411686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.393043 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7"} err="failed to get container status \"b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7\": rpc error: code = NotFound desc = could not find container \"b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7\": container with ID starting with b9153675d39b12c6a1281fbda635ad8526e57f6c90b2082395681de9e57c5bd7 not found: ID does not exist" Nov 01 22:41:14.415223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.413028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:14.441089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.441051 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5"] Nov 01 22:41:14.475661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.473465 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-88896ec5-wtwf5"] Nov 01 22:41:14.490014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.489957 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv"] Nov 01 22:41:14.506465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.506435 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88896ec5-766999f795-pcrhv"] Nov 01 22:41:14.534779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.534733 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:14.546139 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:41:14.546104 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:41:14.741363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.741292 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bad23d-8d92-40ff-a62e-31bcbe226913" path="/var/lib/kubelet/pods/b5bad23d-8d92-40ff-a62e-31bcbe226913/volumes" Nov 01 22:41:14.741846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.741823 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9cd489-b786-4bfb-8536-60b061ce13b2" path="/var/lib/kubelet/pods/bd9cd489-b786-4bfb-8536-60b061ce13b2/volumes" Nov 01 22:41:14.748201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:14.748178 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:15.227649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:15.221862 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54"] Nov 01 22:41:15.227649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:15.225232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:15.390075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:15.390040 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:15.510716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:15.510619 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:16.285936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:16.285905 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:16.454483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:16.454446 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:17.942610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:17.936458 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:17.957497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:17.956695 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:41:17.957497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:17.957079 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:41:17.957782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:17.957511 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:17.965601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:17.963350 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:18.055751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.047906 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.145067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.145018 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:41:18.145699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.145597 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:41:18.170601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.156897 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.308689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.304408 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.550023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.549964 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.657962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.657862 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.686975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.674626 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.688435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.687863 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.968622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.956750 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:41:18.968622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.956879 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:41:18.968622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.957156 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:41:18.968622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.958005 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:41:18.968622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.958064 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:41:18.969483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.954617 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:18.996348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.974890 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:41:18.997987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:18.996660 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.023668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.023637 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:41:19.252654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.197655 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:41:19.252654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.198261 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:41:19.252654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.230960 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:41:19.307602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.282678 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.333603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.296629 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.637656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.330659 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:19.642968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.330698 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:19.661933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.330767 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:41:19.693604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.692771 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:41:19.693604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.692852 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:41:19.693604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.692919 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:41:19.799629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.703119 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:19.991769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.550700 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:41:20.036638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.430644 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.036638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.658328 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:41:20.037162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.663894 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:20.089930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.837610 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:41:20.090433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.964777 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.135539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.964824 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:41:20.136876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:19.964846 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:41:20.142803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.003459 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:41:20.228944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.228834 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:41:20.228944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.228886 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:41:20.228944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.228925 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:41:20.229448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.228957 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:41:20.268736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.003743 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:41:20.282886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.038677 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:20.283184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.283148 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.283386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.213642 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.285074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.004039 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.286838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.228713 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" podUID="d5f8b0f5-9a48-413e-991d-944779679a9d" containerName="payload-processing" containerID="cri-o://20b11c93d855f5c8b32aff087e4a7ea75356fce3f229657845ea87640ac6b046" gracePeriod=25 Nov 01 22:41:20.287091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.233330 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.288826 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.288804 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:41:20.436096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.311662 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:41:20.479633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.476282 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.649884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.646692 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:20.650113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.649910 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:41:20.682919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.670678 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:41:20.794914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.794877 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:20.795370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:20.795342 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.182589 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.182522 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:41:21.234395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.183172 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:41:21.238651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.183266 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.249605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.183291 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:41:21.250242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.209875 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:41:21.281658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.281604 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.295655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.284166 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.295655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.285477 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.325057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.325022 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:21.325461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.325435 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:21.353598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.335352 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:41:21.383597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.365074 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.383802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.377558 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.464613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.438129 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.667892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.667774 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:21.668528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.668494 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:41:21.679907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:21.679869 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.220545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.220511 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:22.250745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.234510 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.251928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.251897 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:41:22.252347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.252323 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.287533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.260204 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.287533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.252690 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:41:22.287533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.285350 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.287533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.285949 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:41:22.287533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.287231 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:41:22.340635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.338542 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.375102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.375046 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.403695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.399464 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:41:22.408908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.408880 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:41:22.453623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.446685 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:41:22.488070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.487972 4981 generic.go:356] "Generic (PLEG): container finished" podID="d5f8b0f5-9a48-413e-991d-944779679a9d" containerID="20b11c93d855f5c8b32aff087e4a7ea75356fce3f229657845ea87640ac6b046" exitCode=0 Nov 01 22:41:22.488313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.488289 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" event={"ID":"d5f8b0f5-9a48-413e-991d-944779679a9d","Type":"ContainerDied","Data":"20b11c93d855f5c8b32aff087e4a7ea75356fce3f229657845ea87640ac6b046"} Nov 01 22:41:22.698784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.698738 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:22.699336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.699286 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:22.699690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:22.699666 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:41:23.045601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.041623 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:23.317724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.282888 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.317724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.296171 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.317724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.307631 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.440439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.323169 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.579692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.373011 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:23.592628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.373095 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.633406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.405116 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:41:23.633406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.405433 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:41:23.633406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.441848 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.633406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.453562 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.633406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.473784 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:41:23.775617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.767465 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:23.775617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.767778 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:23.775617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.768013 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:41:23.854922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:23.854815 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.078635 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:41:24.034382 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.251s" Nov 01 22:41:24.078635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.034545 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:41:24.122719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.111172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:24.304655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.304596 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.522605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.510173 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.545840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.545792 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.584602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.583714 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded" Nov 01 22:41:24.589640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.585017 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.614602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.593263 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.615072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.615032 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.633636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.627257 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:41:24.634895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.629338 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.662954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.632757 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:24.678606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.630716 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:41:24.679293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.647684 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:41:24.828889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.828220 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:24.905638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.905609 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:41:24.949769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.949721 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.059661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.955450 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.059958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.958659 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:41:25.098706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.958688 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:41:25.105608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.958707 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:41:25.105608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.951014 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.105608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:24.969679 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 01 22:41:25.128707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.118260 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f8b0f5-9a48-413e-991d-944779679a9d-kube-api-access-86g8w\" (UniqueName: \"kubernetes.io/projected/d5f8b0f5-9a48-413e-991d-944779679a9d-kube-api-access-86g8w\") pod \"d5f8b0f5-9a48-413e-991d-944779679a9d\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " Nov 01 22:41:25.217078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.180692 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f8b0f5-9a48-413e-991d-944779679a9d-kube-api-access-86g8w" pod "d5f8b0f5-9a48-413e-991d-944779679a9d" (UID: "d5f8b0f5-9a48-413e-991d-944779679a9d"). InnerVolumeSpecName "kube-api-access-86g8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:25.238191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.231865 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f8b0f5-9a48-413e-991d-944779679a9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f8b0f5-9a48-413e-991d-944779679a9d-plugins-config-volume\") pod \"d5f8b0f5-9a48-413e-991d-944779679a9d\" (UID: \"d5f8b0f5-9a48-413e-991d-944779679a9d\") " Nov 01 22:41:25.238191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.232854 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86g8w\" (UniqueName: \"kubernetes.io/projected/d5f8b0f5-9a48-413e-991d-944779679a9d-kube-api-access-86g8w\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:25.238515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.238485 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f8b0f5-9a48-413e-991d-944779679a9d-plugins-config-volume" pod "d5f8b0f5-9a48-413e-991d-944779679a9d" (UID: "d5f8b0f5-9a48-413e-991d-944779679a9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:25.297599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.275658 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:25.297599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.291669 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.322793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.322681 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.440698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.337203 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f8b0f5-9a48-413e-991d-944779679a9d-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:25.650211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.650108 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:41:25.731747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.686735 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:41:25.733334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.733290 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.777279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.687154 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.807620 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:41:25.802233 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.074s" Nov 01 22:41:25.807620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.802509 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:41:25.807620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.802599 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:41:25.807620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.802651 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:41:25.807620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.807053 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" event={"ID":"d5f8b0f5-9a48-413e-991d-944779679a9d","Type":"ContainerDied","Data":"692871c22d26aade24b2c1c77f6510755dca9945fbaf5a964a5a95a329003824"} Nov 01 22:41:25.807620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.807303 4981 scope.go:122] "RemoveContainer" containerID="20b11c93d855f5c8b32aff087e4a7ea75356fce3f229657845ea87640ac6b046" Nov 01 22:41:25.823607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.808205 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54" Nov 01 22:41:25.900148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.897754 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:41:25.911286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.911193 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:41:25.935802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.935210 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:41:25.936900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.936774 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:41:25.937998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.937959 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:41:25.938322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.938302 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:25.939474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.939439 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:41:25.977724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.977686 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.996194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.987889 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:25.996553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.996511 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:25.996762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:25.996737 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:26.055756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.055302 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq"] Nov 01 22:41:26.062868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.062837 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:41:26.064441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.064404 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:41:26.065257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.065221 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:41:26.066093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.066061 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:41:26.114628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.087872 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0f6226e-q7fmq"] Nov 01 22:41:26.118569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.118536 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5"] Nov 01 22:41:26.120868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.120827 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" podUID="e0225bdd-0a3e-4013-9a2d-0623513fe8f3" containerName="istio-proxy" containerID="cri-o://1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac" gracePeriod=30 Nov 01 22:41:26.138961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.138545 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g"] Nov 01 22:41:26.140302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.140255 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d5f8b0f5-9a48-413e-991d-944779679a9d" containerName="payload-processing" Nov 01 22:41:26.140534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.140278 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f8b0f5-9a48-413e-991d-944779679a9d" containerName="payload-processing" Nov 01 22:41:26.140776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.140704 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bd9cd489-b786-4bfb-8536-60b061ce13b2" containerName="payload-processing" Nov 01 22:41:26.140776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.140722 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9cd489-b786-4bfb-8536-60b061ce13b2" containerName="payload-processing" Nov 01 22:41:26.141014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.140974 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b9914a5f-608f-4738-a1ac-78b0930451bf" containerName="revoke-keys" Nov 01 22:41:26.141171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.140991 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9914a5f-608f-4738-a1ac-78b0930451bf" containerName="revoke-keys" Nov 01 22:41:26.141571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.141505 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9914a5f-608f-4738-a1ac-78b0930451bf" containerName="revoke-keys" Nov 01 22:41:26.141571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.141537 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f8b0f5-9a48-413e-991d-944779679a9d" containerName="payload-processing" Nov 01 22:41:26.141886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.141807 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9cd489-b786-4bfb-8536-60b061ce13b2" containerName="payload-processing" Nov 01 22:41:26.149085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.149061 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.155512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.155484 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g"] Nov 01 22:41:26.163256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.163191 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b8b59d8d\"" Nov 01 22:41:26.163745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.163725 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b8b59d8d-dockercfg-fr656\"" Nov 01 22:41:26.167150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.167125 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k"] Nov 01 22:41:26.175814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.173773 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.185610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.184496 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k"] Nov 01 22:41:26.248419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.248382 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d7d076-bca0-45ef-8b71-51359edaf53d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.248678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.248654 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.248761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.248745 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bpt\" (UniqueName: \"kubernetes.io/projected/69d7d076-bca0-45ef-8b71-51359edaf53d-kube-api-access-56bpt\") pod \"payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.248833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.248772 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5894r\" (UniqueName: \"kubernetes.io/projected/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-kube-api-access-5894r\") pod \"payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.271024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.267084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:26.355675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.355641 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d7d076-bca0-45ef-8b71-51359edaf53d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.355958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.355931 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.356200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.356177 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bpt\" (UniqueName: \"kubernetes.io/projected/69d7d076-bca0-45ef-8b71-51359edaf53d-kube-api-access-56bpt\") pod \"payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.356347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.356330 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5894r\" (UniqueName: \"kubernetes.io/projected/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-kube-api-access-5894r\") pod \"payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.357501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.357445 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.360743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.360721 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d7d076-bca0-45ef-8b71-51359edaf53d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.369682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.369641 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bpt\" (UniqueName: \"kubernetes.io/projected/69d7d076-bca0-45ef-8b71-51359edaf53d-kube-api-access-56bpt\") pod \"payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.418173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.418092 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5894r\" (UniqueName: \"kubernetes.io/projected/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-kube-api-access-5894r\") pod \"payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.502600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.486668 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54"] Nov 01 22:41:26.506545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.506023 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e61a2d4e-dd4bf8447-6bk54"] Nov 01 22:41:26.518122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.518087 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:26.552048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.547560 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:26.827385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.826750 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fef3bd5-0401-4b2c-bb69-8ef2acce50a1" path="/var/lib/kubelet/pods/7fef3bd5-0401-4b2c-bb69-8ef2acce50a1/volumes" Nov 01 22:41:26.827858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:26.827478 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f8b0f5-9a48-413e-991d-944779679a9d" path="/var/lib/kubelet/pods/d5f8b0f5-9a48-413e-991d-944779679a9d/volumes" Nov 01 22:41:27.113168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.113072 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:27.128680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.128648 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k"] Nov 01 22:41:27.140627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.139435 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g"] Nov 01 22:41:27.255466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.255368 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:27.395719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.389808 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" event={"ID":"69d7d076-bca0-45ef-8b71-51359edaf53d","Type":"ContainerStarted","Data":"245cd68388f4103ad444b4e9589ae44153031955ac253b7b63f78c79313c7074"} Nov 01 22:41:27.423598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.409015 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" event={"ID":"87aa5dc8-bf4e-40ca-b2b6-d322a0737384","Type":"ContainerStarted","Data":"a52acbc062645fcfa08bc511b543a455ea50435e8eda081394892beeee76455b"} Nov 01 22:41:27.706798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.706711 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:27.731283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.731256 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v"] Nov 01 22:41:27.741387 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.741353 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e61a2d4e-lrw9v"] Nov 01 22:41:27.809543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.809503 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:27.934617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:27.933769 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:28.420118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.420080 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" event={"ID":"69d7d076-bca0-45ef-8b71-51359edaf53d","Type":"ContainerStarted","Data":"9b89ffd9af583414888c8a73f24a57f2a79d0775f81cf226db3fcdade250f0ee"} Nov 01 22:41:28.420312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.420151 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:28.421787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.421757 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" event={"ID":"87aa5dc8-bf4e-40ca-b2b6-d322a0737384","Type":"ContainerStarted","Data":"e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce"} Nov 01 22:41:28.421917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.421814 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:28.451395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.451297 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" podStartSLOduration=4.451284391 podStartE2EDuration="4.451284391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:28.441777169 +0000 UTC m=+1053.869655100" watchObservedRunningTime="2026-11-01 22:41:28.451284391 +0000 UTC m=+1053.879162321" Nov 01 22:41:28.465549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.465502 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" podStartSLOduration=4.465489613 podStartE2EDuration="4.465489613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:28.459038615 +0000 UTC m=+1053.886916547" watchObservedRunningTime="2026-11-01 22:41:28.465489613 +0000 UTC m=+1053.893367544" Nov 01 22:41:28.742461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.742364 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9914a5f-608f-4738-a1ac-78b0930451bf" path="/var/lib/kubelet/pods/b9914a5f-608f-4738-a1ac-78b0930451bf/volumes" Nov 01 22:41:28.819098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.818600 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:28.969216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:28.969179 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:29.082071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:29.081998 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:29.205599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:29.202083 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:29.871437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:29.871402 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:30.234219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.234182 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:30.248768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.248727 4981 patch_prober.go:29] interesting pod/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:41:30.248931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.248795 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" podUID="e0225bdd-0a3e-4013-9a2d-0623513fe8f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:41:30.333116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.333076 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2"] Nov 01 22:41:30.340294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.340272 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:30.351272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.351016 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b07f40e8\"" Nov 01 22:41:30.351514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.351493 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b07f40e8-dockercfg-mxd9f\"" Nov 01 22:41:30.361432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.361403 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2"] Nov 01 22:41:30.389709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.381354 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:30.427388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.427352 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:30.427645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.427608 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvlt\" (UniqueName: \"kubernetes.io/projected/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-kube-api-access-fhvlt\") pod \"payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:30.524650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.514148 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:30.555150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.555103 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:30.557678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.557637 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvlt\" (UniqueName: \"kubernetes.io/projected/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-kube-api-access-fhvlt\") pod \"payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:30.558486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.557387 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:30.635600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.634404 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvlt\" (UniqueName: \"kubernetes.io/projected/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-kube-api-access-fhvlt\") pod \"payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:30.666606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:30.665451 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:31.066629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:31.066599 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2"] Nov 01 22:41:31.356127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:31.356045 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:31.466742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:31.466708 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" event={"ID":"6e6c3ebc-1ed8-45ec-952a-a029fdaab510","Type":"ContainerStarted","Data":"48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c"} Nov 01 22:41:31.466930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:31.466764 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" event={"ID":"6e6c3ebc-1ed8-45ec-952a-a029fdaab510","Type":"ContainerStarted","Data":"b2c8bdba5feb525246fde01e37076a9f831e108482f247aab91777ba1e1e7f4a"} Nov 01 22:41:31.467709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:31.467685 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:31.504791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:31.504724 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" podStartSLOduration=1.504703679 podStartE2EDuration="1.504703679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:31.496804264 +0000 UTC m=+1056.924682212" watchObservedRunningTime="2026-11-01 22:41:31.504703679 +0000 UTC m=+1056.932581610" Nov 01 22:41:32.292392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:32.292362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:33.062860 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.062829 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:33.168479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.168434 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:33.404743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.404713 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:41:33.405654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.405628 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:33.495340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.495304 4981 generic.go:356] "Generic (PLEG): container finished" podID="e0225bdd-0a3e-4013-9a2d-0623513fe8f3" containerID="1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac" exitCode=0 Nov 01 22:41:33.495736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.495705 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" Nov 01 22:41:33.499957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.495735 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" event={"ID":"e0225bdd-0a3e-4013-9a2d-0623513fe8f3","Type":"ContainerDied","Data":"1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac"} Nov 01 22:41:33.499957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.495810 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5" event={"ID":"e0225bdd-0a3e-4013-9a2d-0623513fe8f3","Type":"ContainerDied","Data":"bf4b847ba916ce2a21856399cf7e91519173a1641ada5a3e26565d0b4af16989"} Nov 01 22:41:33.499957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.495861 4981 scope.go:122] "RemoveContainer" containerID="1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac" Nov 01 22:41:33.509324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.509306 4981 scope.go:122] "RemoveContainer" containerID="1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac" Nov 01 22:41:33.509639 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:41:33.509619 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac\": container with ID starting with 1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac not found: ID does not exist" containerID="1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac" Nov 01 22:41:33.509722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.509646 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac"} err="failed to get container status \"1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac\": rpc error: code = NotFound desc = could not find container \"1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac\": container with ID starting with 1a4195b10ef8d915b3558942aeca10c8d54d97d4e59be7a1408017dda47804ac not found: ID does not exist" Nov 01 22:41:33.523215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523190 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-data\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523306 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-envoy\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523334 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-credential-socket\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523381 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-podinfo\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523428 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-data" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:33.523484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523452 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istiod-ca-cert\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523649 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-kube-api-access-bb72q\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-kube-api-access-bb72q\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523668 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-credential-socket" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:33.523717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523683 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-socket\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523731 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-certs\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523771 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istiod-ca-cert" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:33.523908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523855 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-token\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-token\") pod \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\" (UID: \"e0225bdd-0a3e-4013-9a2d-0623513fe8f3\") " Nov 01 22:41:33.523989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.523964 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-socket" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:33.524169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.524146 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-certs" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:33.524772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.524746 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.524874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.524774 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.524874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.524791 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.524874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.524805 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.524874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.524822 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.525496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.525470 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-envoy" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:41:33.525689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.525668 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-podinfo" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:41:33.525907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.525892 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-token" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:33.526001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.525989 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-kube-api-access-bb72q" pod "e0225bdd-0a3e-4013-9a2d-0623513fe8f3" (UID: "e0225bdd-0a3e-4013-9a2d-0623513fe8f3"). InnerVolumeSpecName "kube-api-access-bb72q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:33.636171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.629668 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.636171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.629705 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.636171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.629722 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb72q\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-kube-api-access-bb72q\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.636171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.629738 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0225bdd-0a3e-4013-9a2d-0623513fe8f3-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:33.830866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.830822 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5"] Nov 01 22:41:33.837291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.837264 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0f6226e-openshift-default-5885b6675-b5sl5"] Nov 01 22:41:33.998744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:33.998709 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:34.738365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:34.738333 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0225bdd-0a3e-4013-9a2d-0623513fe8f3" path="/var/lib/kubelet/pods/e0225bdd-0a3e-4013-9a2d-0623513fe8f3/volumes" Nov 01 22:41:34.769254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:34.769228 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:35.622654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:35.622625 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:36.427499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:36.427471 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:36.528152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:36.528122 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:36.634427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:36.634392 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:37.333605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:37.333554 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:37.985678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:37.985634 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:38.850950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:38.850907 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:39.437301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:39.437272 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:41:39.437757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:39.437351 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:41:39.660313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:39.660285 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:40.412396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:40.412346 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:41.052060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:41.052020 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:41.656522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:41.656485 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:42.033514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:42.033486 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc"] Nov 01 22:41:42.040077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:42.040048 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c4fd1c-9gzcc"] Nov 01 22:41:42.284207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:42.284121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:42.736523 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:42.736487 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d7ca94-df4d-4abb-b654-f883952fec10" path="/var/lib/kubelet/pods/b8d7ca94-df4d-4abb-b654-f883952fec10/volumes" Nov 01 22:41:42.972708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:42.972679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:43.499323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:43.499295 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:41:43.807639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:43.807546 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:44.525698 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:41:44.525659 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:41:44.571809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:44.571771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:45.243482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:45.243441 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:45.875996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:45.875963 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:46.651010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:46.650981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:47.286314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:47.286286 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:47.890167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:47.890136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:48.732562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:48.732528 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:49.590514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:49.590480 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:50.225356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:50.225327 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:50.975237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:50.975206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:51.580088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.580054 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4"] Nov 01 22:41:51.580871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.580855 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e0225bdd-0a3e-4013-9a2d-0623513fe8f3" containerName="istio-proxy" Nov 01 22:41:51.580871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.580871 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0225bdd-0a3e-4013-9a2d-0623513fe8f3" containerName="istio-proxy" Nov 01 22:41:51.580990 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.580981 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0225bdd-0a3e-4013-9a2d-0623513fe8f3" containerName="istio-proxy" Nov 01 22:41:51.584888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.584869 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.592068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.592014 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:41:51.592218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.592199 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:41:51.592507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.592256 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:41:51.596680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.592324 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:41:51.623593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.623296 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4"] Nov 01 22:41:51.656292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.656253 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.656803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.656775 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.657051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.657032 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5plt\" (UniqueName: \"kubernetes.io/projected/7f38292f-c9c9-4279-872d-ea75704e608f-kube-api-access-r5plt\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.759562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.759506 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.760270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.760227 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.760520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.760502 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5plt\" (UniqueName: \"kubernetes.io/projected/7f38292f-c9c9-4279-872d-ea75704e608f-kube-api-access-r5plt\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.781872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.780975 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.828727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.821074 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5plt\" (UniqueName: \"kubernetes.io/projected/7f38292f-c9c9-4279-872d-ea75704e608f-kube-api-access-r5plt\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.862144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.862066 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:51.919855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:51.919824 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:52.169456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:52.169366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:52.224818 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:41:52.224772 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f38292f_c9c9_4279_872d_ea75704e608f.slice/crio-a81717d7493872d902fa90614ba9e1af9e45f64d38f237df4ee17ad58461299b WatchSource:0}: Error finding container a81717d7493872d902fa90614ba9e1af9e45f64d38f237df4ee17ad58461299b: Status 404 returned error can't find the container with id a81717d7493872d902fa90614ba9e1af9e45f64d38f237df4ee17ad58461299b Nov 01 22:41:52.247635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:52.247568 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4"] Nov 01 22:41:52.900268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:52.900178 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" event={"ID":"7f38292f-c9c9-4279-872d-ea75704e608f","Type":"ContainerStarted","Data":"6125a205f272a8405f2ce53525cbf19be55df25c5310200405a0203424b0c1b9"} Nov 01 22:41:52.900268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:52.900244 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" event={"ID":"7f38292f-c9c9-4279-872d-ea75704e608f","Type":"ContainerStarted","Data":"a81717d7493872d902fa90614ba9e1af9e45f64d38f237df4ee17ad58461299b"} Nov 01 22:41:52.921825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:52.921770 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" podStartSLOduration=1.9217527269999999 podStartE2EDuration="1.921752727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:52.916843026 +0000 UTC m=+1078.344720959" watchObservedRunningTime="2026-11-01 22:41:52.921752727 +0000 UTC m=+1078.349630662" Nov 01 22:41:52.966759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:52.966729 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:53.744395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:53.744350 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:53.913420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:53.913374 4981 generic.go:356] "Generic (PLEG): container finished" podID="7f38292f-c9c9-4279-872d-ea75704e608f" containerID="6125a205f272a8405f2ce53525cbf19be55df25c5310200405a0203424b0c1b9" exitCode=0 Nov 01 22:41:53.913977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:53.913491 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" event={"ID":"7f38292f-c9c9-4279-872d-ea75704e608f","Type":"ContainerDied","Data":"6125a205f272a8405f2ce53525cbf19be55df25c5310200405a0203424b0c1b9"} Nov 01 22:41:54.031513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.031419 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch"] Nov 01 22:41:54.037850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.037814 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf67a2df-8qlch"] Nov 01 22:41:54.362360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.362274 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:54.736827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.736796 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab" path="/var/lib/kubelet/pods/73cff08d-8fcb-4dd1-bbe9-9077d66ca8ab/volumes" Nov 01 22:41:54.785380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.785352 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf"] Nov 01 22:41:54.793633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.791400 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.797133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.797107 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf"] Nov 01 22:41:54.822681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.822648 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.822847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.822741 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.822914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.822884 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbrs\" (UniqueName: \"kubernetes.io/projected/0c15ae7f-dd13-423f-9118-a20bcca6fce4-kube-api-access-jcbrs\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.925892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.925862 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.926301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.925924 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.926301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.925988 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbrs\" (UniqueName: \"kubernetes.io/projected/0c15ae7f-dd13-423f-9118-a20bcca6fce4-kube-api-access-jcbrs\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.926783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.926763 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.928586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.928552 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:54.936372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:54.936342 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbrs\" (UniqueName: \"kubernetes.io/projected/0c15ae7f-dd13-423f-9118-a20bcca6fce4-kube-api-access-jcbrs\") pod \"maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:55.115619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.115588 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:55.128205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.128182 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:55.230414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.230388 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-openshift-ca\") pod \"7f38292f-c9c9-4279-872d-ea75704e608f\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " Nov 01 22:41:55.230565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.230541 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f38292f-c9c9-4279-872d-ea75704e608f-kube-api-access-r5plt\" (UniqueName: \"kubernetes.io/projected/7f38292f-c9c9-4279-872d-ea75704e608f-kube-api-access-r5plt\") pod \"7f38292f-c9c9-4279-872d-ea75704e608f\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " Nov 01 22:41:55.230878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.230676 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-xks-ca\") pod \"7f38292f-c9c9-4279-872d-ea75704e608f\" (UID: \"7f38292f-c9c9-4279-872d-ea75704e608f\") " Nov 01 22:41:55.231476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.231114 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-openshift-ca" pod "7f38292f-c9c9-4279-872d-ea75704e608f" (UID: "7f38292f-c9c9-4279-872d-ea75704e608f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:55.231476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.231450 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:55.233281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.233259 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f38292f-c9c9-4279-872d-ea75704e608f-kube-api-access-r5plt" pod "7f38292f-c9c9-4279-872d-ea75704e608f" (UID: "7f38292f-c9c9-4279-872d-ea75704e608f"). InnerVolumeSpecName "kube-api-access-r5plt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:55.233955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.233926 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-xks-ca" pod "7f38292f-c9c9-4279-872d-ea75704e608f" (UID: "7f38292f-c9c9-4279-872d-ea75704e608f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:55.334040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.333951 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5plt\" (UniqueName: \"kubernetes.io/projected/7f38292f-c9c9-4279-872d-ea75704e608f-kube-api-access-r5plt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:55.334040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.333983 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f38292f-c9c9-4279-872d-ea75704e608f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:55.392908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.392878 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:55.432928 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:41:55.432897 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c15ae7f_dd13_423f_9118_a20bcca6fce4.slice/crio-8276d0fcb96d081da05ffc6609d172ceb46d987944feb340d3735a36a7f44c80 WatchSource:0}: Error finding container 8276d0fcb96d081da05ffc6609d172ceb46d987944feb340d3735a36a7f44c80: Status 404 returned error can't find the container with id 8276d0fcb96d081da05ffc6609d172ceb46d987944feb340d3735a36a7f44c80 Nov 01 22:41:55.435018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.434992 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf"] Nov 01 22:41:55.934392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.934360 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" Nov 01 22:41:55.934830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.934368 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4" event={"ID":"7f38292f-c9c9-4279-872d-ea75704e608f","Type":"ContainerDied","Data":"a81717d7493872d902fa90614ba9e1af9e45f64d38f237df4ee17ad58461299b"} Nov 01 22:41:55.934830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.934497 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a81717d7493872d902fa90614ba9e1af9e45f64d38f237df4ee17ad58461299b" Nov 01 22:41:55.936185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.936160 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" event={"ID":"0c15ae7f-dd13-423f-9118-a20bcca6fce4","Type":"ContainerStarted","Data":"9fdc7bd9b985c6429aa8c7678da9067fd10990f1a10a67f7093d8b391272b082"} Nov 01 22:41:55.936311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.936208 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" event={"ID":"0c15ae7f-dd13-423f-9118-a20bcca6fce4","Type":"ContainerStarted","Data":"8276d0fcb96d081da05ffc6609d172ceb46d987944feb340d3735a36a7f44c80"} Nov 01 22:41:55.962083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:55.962040 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" podStartSLOduration=1.962025409 podStartE2EDuration="1.962025409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:55.956707063 +0000 UTC m=+1081.384585001" watchObservedRunningTime="2026-11-01 22:41:55.962025409 +0000 UTC m=+1081.389903339" Nov 01 22:41:56.027793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.027757 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj"] Nov 01 22:41:56.034365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.034336 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l"] Nov 01 22:41:56.039936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.039915 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155961b8-gsblj"] Nov 01 22:41:56.045457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.045437 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c9b0f2d-jbt7l"] Nov 01 22:41:56.077693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.077669 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:56.254667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.254639 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:56.735789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.735756 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf" path="/var/lib/kubelet/pods/0d5cf966-5553-4fe5-bee3-e4a0c8c5d7cf/volumes" Nov 01 22:41:56.736241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.736227 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7" path="/var/lib/kubelet/pods/c6b8ae50-f70e-464b-95fc-d68bc5f8d7d7/volumes" Nov 01 22:41:56.870756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.870708 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:56.957136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.957098 4981 generic.go:356] "Generic (PLEG): container finished" podID="0c15ae7f-dd13-423f-9118-a20bcca6fce4" containerID="9fdc7bd9b985c6429aa8c7678da9067fd10990f1a10a67f7093d8b391272b082" exitCode=0 Nov 01 22:41:56.957610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:56.957192 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" event={"ID":"0c15ae7f-dd13-423f-9118-a20bcca6fce4","Type":"ContainerDied","Data":"9fdc7bd9b985c6429aa8c7678da9067fd10990f1a10a67f7093d8b391272b082"} Nov 01 22:41:57.627233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:57.627155 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:58.042517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.042471 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf"] Nov 01 22:41:58.053630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.053599 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4"] Nov 01 22:41:58.063045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.063015 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f9c635-fr4nf"] Nov 01 22:41:58.069084 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.069060 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cfa5fa7-npmx4"] Nov 01 22:41:58.147467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.147442 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:58.280325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.280231 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-xks-ca\") pod \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " Nov 01 22:41:58.280497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.280405 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-openshift-ca\") pod \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " Nov 01 22:41:58.280497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.280426 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c15ae7f-dd13-423f-9118-a20bcca6fce4-kube-api-access-jcbrs\" (UniqueName: \"kubernetes.io/projected/0c15ae7f-dd13-423f-9118-a20bcca6fce4-kube-api-access-jcbrs\") pod \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\" (UID: \"0c15ae7f-dd13-423f-9118-a20bcca6fce4\") " Nov 01 22:41:58.280817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.280788 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-openshift-ca" pod "0c15ae7f-dd13-423f-9118-a20bcca6fce4" (UID: "0c15ae7f-dd13-423f-9118-a20bcca6fce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:41:58.281234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.281212 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:58.282483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.282458 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-xks-ca" pod "0c15ae7f-dd13-423f-9118-a20bcca6fce4" (UID: "0c15ae7f-dd13-423f-9118-a20bcca6fce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:41:58.282647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.282522 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c15ae7f-dd13-423f-9118-a20bcca6fce4-kube-api-access-jcbrs" pod "0c15ae7f-dd13-423f-9118-a20bcca6fce4" (UID: "0c15ae7f-dd13-423f-9118-a20bcca6fce4"). InnerVolumeSpecName "kube-api-access-jcbrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:41:58.341191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.341164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:58.389718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.383341 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcbrs\" (UniqueName: \"kubernetes.io/projected/0c15ae7f-dd13-423f-9118-a20bcca6fce4-kube-api-access-jcbrs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:58.389718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.383377 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c15ae7f-dd13-423f-9118-a20bcca6fce4-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:41:58.738106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.738069 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f8a78a-22ff-4d14-825d-e1d63f4d8a8e" path="/var/lib/kubelet/pods/72f8a78a-22ff-4d14-825d-e1d63f4d8a8e/volumes" Nov 01 22:41:58.738637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.738623 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0ad2d7-1848-479c-9ece-29970c872a7e" path="/var/lib/kubelet/pods/fb0ad2d7-1848-479c-9ece-29970c872a7e/volumes" Nov 01 22:41:58.972601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.972555 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:41:58.981694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.981666 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" Nov 01 22:41:58.981818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.981695 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf" event={"ID":"0c15ae7f-dd13-423f-9118-a20bcca6fce4","Type":"ContainerDied","Data":"8276d0fcb96d081da05ffc6609d172ceb46d987944feb340d3735a36a7f44c80"} Nov 01 22:41:58.981818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:58.981746 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8276d0fcb96d081da05ffc6609d172ceb46d987944feb340d3735a36a7f44c80" Nov 01 22:41:59.610816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:41:59.610779 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:00.270173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:00.270140 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:00.979372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:00.979341 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:01.671336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:01.671292 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:01.794042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:01.794004 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k"] Nov 01 22:42:01.812551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:01.812501 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g"] Nov 01 22:42:01.858034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:01.858004 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:01.953725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:01.953687 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:02.579902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:02.579862 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:02.682043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:02.682008 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:02.790565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:02.790534 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:03.239407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:03.239372 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:03.420596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:03.411026 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:03.504382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:03.504267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:03.611188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:03.611158 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4"] Nov 01 22:42:03.619415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:03.619376 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b8b59d8d-57fl4"] Nov 01 22:42:03.775144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:03.775028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:03.916541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:03.916507 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:04.638282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:04.638246 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:04.736694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:04.736662 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f38292f-c9c9-4279-872d-ea75704e608f" path="/var/lib/kubelet/pods/7f38292f-c9c9-4279-872d-ea75704e608f/volumes" Nov 01 22:42:04.963114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:04.963075 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:05.153685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:05.153636 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:05.276187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:05.276096 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2"] Nov 01 22:42:05.296566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:05.296530 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:06.022245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:06.022190 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:06.382765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:06.382651 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:06.795122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:06.795080 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" podUID="87aa5dc8-bf4e-40ca-b2b6-d322a0737384" containerName="payload-processing" containerID="cri-o://e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce" gracePeriod=25 Nov 01 22:42:06.813857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:06.813806 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" podUID="69d7d076-bca0-45ef-8b71-51359edaf53d" containerName="payload-pre-processing" containerID="cri-o://9b89ffd9af583414888c8a73f24a57f2a79d0775f81cf226db3fcdade250f0ee" gracePeriod=25 Nov 01 22:42:07.062916 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.062786 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85"] Nov 01 22:42:07.078609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.078547 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26"] Nov 01 22:42:07.087680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.087630 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a46cba-9js85"] Nov 01 22:42:07.099051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.099020 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a504e80c-6dv26"] Nov 01 22:42:07.143930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.143871 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:42:07.235262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.228407 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:07.258414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.258361 4981 generic.go:356] "Generic (PLEG): container finished" podID="69d7d076-bca0-45ef-8b71-51359edaf53d" containerID="9b89ffd9af583414888c8a73f24a57f2a79d0775f81cf226db3fcdade250f0ee" exitCode=0 Nov 01 22:42:07.258556 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.258498 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" event={"ID":"69d7d076-bca0-45ef-8b71-51359edaf53d","Type":"ContainerDied","Data":"9b89ffd9af583414888c8a73f24a57f2a79d0775f81cf226db3fcdade250f0ee"} Nov 01 22:42:07.260561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.260534 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:42:07.262040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.262003 4981 generic.go:356] "Generic (PLEG): container finished" podID="87aa5dc8-bf4e-40ca-b2b6-d322a0737384" containerID="e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce" exitCode=0 Nov 01 22:42:07.262379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.262353 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" event={"ID":"87aa5dc8-bf4e-40ca-b2b6-d322a0737384","Type":"ContainerDied","Data":"e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce"} Nov 01 22:42:07.263366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.263339 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" event={"ID":"87aa5dc8-bf4e-40ca-b2b6-d322a0737384","Type":"ContainerDied","Data":"a52acbc062645fcfa08bc511b543a455ea50435e8eda081394892beeee76455b"} Nov 01 22:42:07.263538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.262715 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k" Nov 01 22:42:07.263645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.263386 4981 scope.go:122] "RemoveContainer" containerID="e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce" Nov 01 22:42:07.284060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.281870 4981 scope.go:122] "RemoveContainer" containerID="e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce" Nov 01 22:42:07.284788 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:42:07.284756 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce\": container with ID starting with e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce not found: ID does not exist" containerID="e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce" Nov 01 22:42:07.284888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.284801 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce"} err="failed to get container status \"e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce\": rpc error: code = NotFound desc = could not find container \"e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce\": container with ID starting with e8e05145a30f164324198e08d68255d654c2f226e42f781c84f062ba43dbb9ce not found: ID does not exist" Nov 01 22:42:07.300207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.300169 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-plugins-config-volume\") pod \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " Nov 01 22:42:07.300562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.300538 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-kube-api-access-5894r\" (UniqueName: \"kubernetes.io/projected/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-kube-api-access-5894r\") pod \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\" (UID: \"87aa5dc8-bf4e-40ca-b2b6-d322a0737384\") " Nov 01 22:42:07.310552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.310520 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-kube-api-access-5894r" pod "87aa5dc8-bf4e-40ca-b2b6-d322a0737384" (UID: "87aa5dc8-bf4e-40ca-b2b6-d322a0737384"). InnerVolumeSpecName "kube-api-access-5894r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:07.310898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.310877 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-plugins-config-volume" pod "87aa5dc8-bf4e-40ca-b2b6-d322a0737384" (UID: "87aa5dc8-bf4e-40ca-b2b6-d322a0737384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:07.361795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.361700 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:07.414598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.410565 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d7d076-bca0-45ef-8b71-51359edaf53d-kube-api-access-56bpt\" (UniqueName: \"kubernetes.io/projected/69d7d076-bca0-45ef-8b71-51359edaf53d-kube-api-access-56bpt\") pod \"69d7d076-bca0-45ef-8b71-51359edaf53d\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " Nov 01 22:42:07.414598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.410777 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d7d076-bca0-45ef-8b71-51359edaf53d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d7d076-bca0-45ef-8b71-51359edaf53d-plugins-config-volume\") pod \"69d7d076-bca0-45ef-8b71-51359edaf53d\" (UID: \"69d7d076-bca0-45ef-8b71-51359edaf53d\") " Nov 01 22:42:07.414598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.411736 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:07.414598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.411761 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5894r\" (UniqueName: \"kubernetes.io/projected/87aa5dc8-bf4e-40ca-b2b6-d322a0737384-kube-api-access-5894r\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:07.415411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.415365 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d7d076-bca0-45ef-8b71-51359edaf53d-plugins-config-volume" pod "69d7d076-bca0-45ef-8b71-51359edaf53d" (UID: "69d7d076-bca0-45ef-8b71-51359edaf53d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:07.419208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.419179 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d7d076-bca0-45ef-8b71-51359edaf53d-kube-api-access-56bpt" pod "69d7d076-bca0-45ef-8b71-51359edaf53d" (UID: "69d7d076-bca0-45ef-8b71-51359edaf53d"). InnerVolumeSpecName "kube-api-access-56bpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:07.522631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.517883 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bpt\" (UniqueName: \"kubernetes.io/projected/69d7d076-bca0-45ef-8b71-51359edaf53d-kube-api-access-56bpt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:07.522631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.517913 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69d7d076-bca0-45ef-8b71-51359edaf53d-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:07.610449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.610408 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k"] Nov 01 22:42:07.617853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.617781 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b8b59d8d-f7f679c45-thj9k"] Nov 01 22:42:07.620302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.620270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:07.695217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.695169 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf"] Nov 01 22:42:07.701339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.701311 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b07f40e8-fxxbf"] Nov 01 22:42:07.928942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:07.928913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:08.277225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.277144 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" Nov 01 22:42:08.277225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.277176 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g" event={"ID":"69d7d076-bca0-45ef-8b71-51359edaf53d","Type":"ContainerDied","Data":"245cd68388f4103ad444b4e9589ae44153031955ac253b7b63f78c79313c7074"} Nov 01 22:42:08.278043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.277236 4981 scope.go:122] "RemoveContainer" containerID="9b89ffd9af583414888c8a73f24a57f2a79d0775f81cf226db3fcdade250f0ee" Nov 01 22:42:08.297707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.297685 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:08.311704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.311674 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g"] Nov 01 22:42:08.318050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.318017 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b8b59d8d-6bc5b5c9c9-6654g"] Nov 01 22:42:08.736643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.736613 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c15ae7f-dd13-423f-9118-a20bcca6fce4" path="/var/lib/kubelet/pods/0c15ae7f-dd13-423f-9118-a20bcca6fce4/volumes" Nov 01 22:42:08.737252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.737234 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d63c56-fb4c-4dcd-b000-59dd90860843" path="/var/lib/kubelet/pods/38d63c56-fb4c-4dcd-b000-59dd90860843/volumes" Nov 01 22:42:08.737697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.737683 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d7d076-bca0-45ef-8b71-51359edaf53d" path="/var/lib/kubelet/pods/69d7d076-bca0-45ef-8b71-51359edaf53d/volumes" Nov 01 22:42:08.738377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.738362 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87aa5dc8-bf4e-40ca-b2b6-d322a0737384" path="/var/lib/kubelet/pods/87aa5dc8-bf4e-40ca-b2b6-d322a0737384/volumes" Nov 01 22:42:08.738832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:08.738817 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccf430b-df53-4283-af3a-c0afd45dd841" path="/var/lib/kubelet/pods/bccf430b-df53-4283-af3a-c0afd45dd841/volumes" Nov 01 22:42:09.023056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.022941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:09.032276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.032123 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7"] Nov 01 22:42:09.038705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.038677 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw"] Nov 01 22:42:09.044113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.044095 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1642c62e-czpt7"] Nov 01 22:42:09.049172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.049150 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e341f3ef-rgjdw"] Nov 01 22:42:09.127519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.127487 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:09.239100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.239060 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:09.958271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:09.958237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:10.277173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.277098 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" podUID="6e6c3ebc-1ed8-45ec-952a-a029fdaab510" containerName="payload-pre-processing" containerID="cri-o://48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c" gracePeriod=25 Nov 01 22:42:10.334895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.334865 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:10.517775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.517744 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:10.559432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.559359 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp"] Nov 01 22:42:10.560696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560568 4981 cpu_manager.go:414] "containerMap: removing container" podUID="87aa5dc8-bf4e-40ca-b2b6-d322a0737384" containerName="payload-processing" Nov 01 22:42:10.560696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560613 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87aa5dc8-bf4e-40ca-b2b6-d322a0737384" containerName="payload-processing" Nov 01 22:42:10.560696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560648 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7f38292f-c9c9-4279-872d-ea75704e608f" containerName="revoke-keys" Nov 01 22:42:10.560696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560656 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f38292f-c9c9-4279-872d-ea75704e608f" containerName="revoke-keys" Nov 01 22:42:10.560696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560685 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0c15ae7f-dd13-423f-9118-a20bcca6fce4" containerName="revoke-keys" Nov 01 22:42:10.560696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560692 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c15ae7f-dd13-423f-9118-a20bcca6fce4" containerName="revoke-keys" Nov 01 22:42:10.560947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560704 4981 cpu_manager.go:414] "containerMap: removing container" podUID="69d7d076-bca0-45ef-8b71-51359edaf53d" containerName="payload-pre-processing" Nov 01 22:42:10.560947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.560711 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d7d076-bca0-45ef-8b71-51359edaf53d" containerName="payload-pre-processing" Nov 01 22:42:10.561253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.561235 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="87aa5dc8-bf4e-40ca-b2b6-d322a0737384" containerName="payload-processing" Nov 01 22:42:10.561300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.561264 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f38292f-c9c9-4279-872d-ea75704e608f" containerName="revoke-keys" Nov 01 22:42:10.561300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.561278 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c15ae7f-dd13-423f-9118-a20bcca6fce4" containerName="revoke-keys" Nov 01 22:42:10.561300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.561293 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d7d076-bca0-45ef-8b71-51359edaf53d" containerName="payload-pre-processing" Nov 01 22:42:10.566056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.566037 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.588008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.587979 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-7366bf-dockercfg-87zwd\"" Nov 01 22:42:10.594188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.589099 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-7366bf\"" Nov 01 22:42:10.616737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.616704 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ac8ace-f63d-4fb0-b642-5653bde50da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.616899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.616804 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6jd\" (UniqueName: \"kubernetes.io/projected/81ac8ace-f63d-4fb0-b642-5653bde50da7-kube-api-access-8c6jd\") pod \"payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.625781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.625669 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp"] Nov 01 22:42:10.636429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.636403 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh"] Nov 01 22:42:10.643515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.643490 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:10.660186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.660161 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh"] Nov 01 22:42:10.661986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.661965 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:42:10.681537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.681506 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:10.717881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.717856 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-plugins-config-volume\") pod \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " Nov 01 22:42:10.718075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.718057 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-kube-api-access-fhvlt\" (UniqueName: \"kubernetes.io/projected/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-kube-api-access-fhvlt\") pod \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\" (UID: \"6e6c3ebc-1ed8-45ec-952a-a029fdaab510\") " Nov 01 22:42:10.718293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.718269 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-plugins-config-volume" pod "6e6c3ebc-1ed8-45ec-952a-a029fdaab510" (UID: "6e6c3ebc-1ed8-45ec-952a-a029fdaab510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:10.718474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.718451 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fml\" (UniqueName: \"kubernetes.io/projected/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-kube-api-access-d5fml\") pod \"payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:10.718771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.718749 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ac8ace-f63d-4fb0-b642-5653bde50da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.718856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.718821 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:10.718939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.718856 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6jd\" (UniqueName: \"kubernetes.io/projected/81ac8ace-f63d-4fb0-b642-5653bde50da7-kube-api-access-8c6jd\") pod \"payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.718999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.718978 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:10.719520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.719497 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ac8ace-f63d-4fb0-b642-5653bde50da7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.720359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.720330 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-kube-api-access-fhvlt" pod "6e6c3ebc-1ed8-45ec-952a-a029fdaab510" (UID: "6e6c3ebc-1ed8-45ec-952a-a029fdaab510"). InnerVolumeSpecName "kube-api-access-fhvlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:10.738987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.738964 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b14fec-8347-4ba2-87fe-6972567bac98" path="/var/lib/kubelet/pods/66b14fec-8347-4ba2-87fe-6972567bac98/volumes" Nov 01 22:42:10.739427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.739414 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3b7e13-11b1-4b30-99c7-7ea9c817a290" path="/var/lib/kubelet/pods/9a3b7e13-11b1-4b30-99c7-7ea9c817a290/volumes" Nov 01 22:42:10.742948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.742918 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6jd\" (UniqueName: \"kubernetes.io/projected/81ac8ace-f63d-4fb0-b642-5653bde50da7-kube-api-access-8c6jd\") pod \"payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.932646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.929352 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:10.954635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.936826 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:10.954635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.937004 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fml\" (UniqueName: \"kubernetes.io/projected/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-kube-api-access-d5fml\") pod \"payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:10.954635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.937481 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvlt\" (UniqueName: \"kubernetes.io/projected/6e6c3ebc-1ed8-45ec-952a-a029fdaab510-kube-api-access-fhvlt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:10.954635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.938896 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:10.974596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.973345 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fml\" (UniqueName: \"kubernetes.io/projected/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-kube-api-access-d5fml\") pod \"payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:10.975548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:10.975522 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:11.062417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.062375 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq"] Nov 01 22:42:11.075162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.074341 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m"] Nov 01 22:42:11.086175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.085552 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9e8db3e-sc8cq"] Nov 01 22:42:11.102914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.097533 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088f6ba-55k9m"] Nov 01 22:42:11.343641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.340159 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp"] Nov 01 22:42:11.377158 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:42:11.374819 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbf0e48_9d53_4450_9584_5969ecc8c1b5.slice/crio-64d9e61b91f31352c65bf3f2953b81a29833a647f7c4d5c6536f14626f8e6f67 WatchSource:0}: Error finding container 64d9e61b91f31352c65bf3f2953b81a29833a647f7c4d5c6536f14626f8e6f67: Status 404 returned error can't find the container with id 64d9e61b91f31352c65bf3f2953b81a29833a647f7c4d5c6536f14626f8e6f67 Nov 01 22:42:11.377234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.377157 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh"] Nov 01 22:42:11.414144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.414114 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" event={"ID":"cdbf0e48-9d53-4450-9584-5969ecc8c1b5","Type":"ContainerStarted","Data":"64d9e61b91f31352c65bf3f2953b81a29833a647f7c4d5c6536f14626f8e6f67"} Nov 01 22:42:11.416051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.416025 4981 generic.go:356] "Generic (PLEG): container finished" podID="6e6c3ebc-1ed8-45ec-952a-a029fdaab510" containerID="48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c" exitCode=0 Nov 01 22:42:11.416165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.416062 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" event={"ID":"6e6c3ebc-1ed8-45ec-952a-a029fdaab510","Type":"ContainerDied","Data":"48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c"} Nov 01 22:42:11.416165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.416101 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" Nov 01 22:42:11.416165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.416121 4981 scope.go:122] "RemoveContainer" containerID="48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c" Nov 01 22:42:11.416309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.416106 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2" event={"ID":"6e6c3ebc-1ed8-45ec-952a-a029fdaab510","Type":"ContainerDied","Data":"b2c8bdba5feb525246fde01e37076a9f831e108482f247aab91777ba1e1e7f4a"} Nov 01 22:42:11.417854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.417826 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" event={"ID":"81ac8ace-f63d-4fb0-b642-5653bde50da7","Type":"ContainerStarted","Data":"25f8dfdc894c08c876b5e21829e2b6d3633bb05071e3d2a05335971da28822fa"} Nov 01 22:42:11.439440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.439417 4981 scope.go:122] "RemoveContainer" containerID="48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c" Nov 01 22:42:11.439748 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:42:11.439723 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c\": container with ID starting with 48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c not found: ID does not exist" containerID="48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c" Nov 01 22:42:11.439845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.439760 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c"} err="failed to get container status \"48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c\": rpc error: code = NotFound desc = could not find container \"48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c\": container with ID starting with 48c9cf72f2fd50dd813626f5522700d9bc1bf6070997b3ea6f4e85c9aab4917c not found: ID does not exist" Nov 01 22:42:11.461305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.461282 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2"] Nov 01 22:42:11.473396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:11.473371 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b07f40e8-97c89bd87-9rnd2"] Nov 01 22:42:12.110949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.110916 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:12.435348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.435301 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" event={"ID":"81ac8ace-f63d-4fb0-b642-5653bde50da7","Type":"ContainerStarted","Data":"f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8"} Nov 01 22:42:12.435657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.435387 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:12.437289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.437257 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" event={"ID":"cdbf0e48-9d53-4450-9584-5969ecc8c1b5","Type":"ContainerStarted","Data":"171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9"} Nov 01 22:42:12.437538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.437507 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:12.460238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.460184 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" podStartSLOduration=2.460171257 podStartE2EDuration="2.460171257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:12.45262317 +0000 UTC m=+1097.880501101" watchObservedRunningTime="2026-11-01 22:42:12.460171257 +0000 UTC m=+1097.888049189" Nov 01 22:42:12.477710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.477658 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" podStartSLOduration=2.4776436410000002 podStartE2EDuration="2.477643641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:12.470400672 +0000 UTC m=+1097.898278604" watchObservedRunningTime="2026-11-01 22:42:12.477643641 +0000 UTC m=+1097.905521572" Nov 01 22:42:12.738059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.737987 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c4bf50-afa1-42ed-b79b-d4e24aaf9196" path="/var/lib/kubelet/pods/43c4bf50-afa1-42ed-b79b-d4e24aaf9196/volumes" Nov 01 22:42:12.738514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.738498 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a4bc7e-0f36-4c30-98e4-d5342ee24b79" path="/var/lib/kubelet/pods/67a4bc7e-0f36-4c30-98e4-d5342ee24b79/volumes" Nov 01 22:42:12.739031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:12.739014 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e6c3ebc-1ed8-45ec-952a-a029fdaab510" path="/var/lib/kubelet/pods/6e6c3ebc-1ed8-45ec-952a-a029fdaab510/volumes" Nov 01 22:42:13.085323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:13.085242 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:13.464868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:13.464837 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:13.600130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:13.600099 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:13.769077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:13.769000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:14.526483 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:42:14.526442 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:42:14.587987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.587958 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:14.831746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.831662 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp"] Nov 01 22:42:14.832473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.832452 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6e6c3ebc-1ed8-45ec-952a-a029fdaab510" containerName="payload-pre-processing" Nov 01 22:42:14.832598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.832475 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e6c3ebc-1ed8-45ec-952a-a029fdaab510" containerName="payload-pre-processing" Nov 01 22:42:14.832720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.832706 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e6c3ebc-1ed8-45ec-952a-a029fdaab510" containerName="payload-pre-processing" Nov 01 22:42:14.837922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.837901 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:14.844155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.843755 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ee07ad\"" Nov 01 22:42:14.874066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.874027 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp"] Nov 01 22:42:14.921545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.921503 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5gbv\" (UniqueName: \"kubernetes.io/projected/8bc2e5c9-6387-494d-ae94-f81de624e0f2-kube-api-access-f5gbv\") pod \"payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:14.921986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.921813 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc2e5c9-6387-494d-ae94-f81de624e0f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:14.927875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.927853 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd"] Nov 01 22:42:14.932713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.932698 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:14.939428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:14.939404 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ee07ad-dockercfg-594rh\"" Nov 01 22:42:15.140657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.130144 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428b87e-8632-484b-a440-73a44e8aec59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:15.148047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.148008 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh9zg\" (UniqueName: \"kubernetes.io/projected/1428b87e-8632-484b-a440-73a44e8aec59-kube-api-access-gh9zg\") pod \"payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:15.157433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.148391 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5gbv\" (UniqueName: \"kubernetes.io/projected/8bc2e5c9-6387-494d-ae94-f81de624e0f2-kube-api-access-f5gbv\") pod \"payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:15.158048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.158015 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc2e5c9-6387-494d-ae94-f81de624e0f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:15.159797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.159750 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc2e5c9-6387-494d-ae94-f81de624e0f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:15.174492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.174462 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd"] Nov 01 22:42:15.188986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.188911 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5gbv\" (UniqueName: \"kubernetes.io/projected/8bc2e5c9-6387-494d-ae94-f81de624e0f2-kube-api-access-f5gbv\") pod \"payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:15.260452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.260411 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428b87e-8632-484b-a440-73a44e8aec59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:15.260640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.260510 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh9zg\" (UniqueName: \"kubernetes.io/projected/1428b87e-8632-484b-a440-73a44e8aec59-kube-api-access-gh9zg\") pod \"payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:15.261696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.261651 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428b87e-8632-484b-a440-73a44e8aec59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:15.270397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.270363 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh9zg\" (UniqueName: \"kubernetes.io/projected/1428b87e-8632-484b-a440-73a44e8aec59-kube-api-access-gh9zg\") pod \"payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:15.460234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.460205 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:15.548972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.548761 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:15.727484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.727382 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:15.785714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.785684 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp"] Nov 01 22:42:15.856666 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:42:15.856631 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1428b87e_8632_484b_a440_73a44e8aec59.slice/crio-9c7a966ac7b529b74b1f82d71a5e095a6b7e654eecf3bd03c3139b68a5edf7b6 WatchSource:0}: Error finding container 9c7a966ac7b529b74b1f82d71a5e095a6b7e654eecf3bd03c3139b68a5edf7b6: Status 404 returned error can't find the container with id 9c7a966ac7b529b74b1f82d71a5e095a6b7e654eecf3bd03c3139b68a5edf7b6 Nov 01 22:42:15.858847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:15.858823 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd"] Nov 01 22:42:16.478057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.478021 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" event={"ID":"8bc2e5c9-6387-494d-ae94-f81de624e0f2","Type":"ContainerStarted","Data":"02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4"} Nov 01 22:42:16.478262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.478079 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" event={"ID":"8bc2e5c9-6387-494d-ae94-f81de624e0f2","Type":"ContainerStarted","Data":"4496c061e506a46368dd23cf2c1a18e4cc006fc6d6000099a60aca66bc13ce59"} Nov 01 22:42:16.478262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.478127 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:16.479997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.479970 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" event={"ID":"1428b87e-8632-484b-a440-73a44e8aec59","Type":"ContainerStarted","Data":"ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28"} Nov 01 22:42:16.480113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.480017 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" event={"ID":"1428b87e-8632-484b-a440-73a44e8aec59","Type":"ContainerStarted","Data":"9c7a966ac7b529b74b1f82d71a5e095a6b7e654eecf3bd03c3139b68a5edf7b6"} Nov 01 22:42:16.480113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.480045 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:16.502947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.502898 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" podStartSLOduration=2.50288422 podStartE2EDuration="2.50288422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:16.498206713 +0000 UTC m=+1101.926084644" watchObservedRunningTime="2026-11-01 22:42:16.50288422 +0000 UTC m=+1101.930762151" Nov 01 22:42:16.516833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.516777 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" podStartSLOduration=2.516764392 podStartE2EDuration="2.516764392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:16.515665189 +0000 UTC m=+1101.943543120" watchObservedRunningTime="2026-11-01 22:42:16.516764392 +0000 UTC m=+1101.944642324" Nov 01 22:42:16.742517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:16.742419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:17.474673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:17.474627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:18.164049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:18.164003 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:18.269517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:18.264663 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:18.409487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:18.409458 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:19.030633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:19.030593 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6"] Nov 01 22:42:19.036360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:19.036330 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573d542f-r88v6"] Nov 01 22:42:19.068012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:19.067985 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:19.643074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:19.643036 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:19.745022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:19.744987 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:19.853595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:19.853435 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:20.660809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:20.660771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:20.756996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:20.756964 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6cdf03-19a1-435a-9268-a17091fdbc01" path="/var/lib/kubelet/pods/dc6cdf03-19a1-435a-9268-a17091fdbc01/volumes" Nov 01 22:42:21.046777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:21.046741 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9"] Nov 01 22:42:21.052299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:21.052277 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba2d1b5-dvpx9"] Nov 01 22:42:21.526031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:21.525989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:21.631435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:21.631387 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:21.731155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:21.731121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:22.514416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:22.514373 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:22.736592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:22.736548 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc47d047-7b84-49b0-9096-dc2b9b9b8989" path="/var/lib/kubelet/pods/dc47d047-7b84-49b0-9096-dc2b9b9b8989/volumes" Nov 01 22:42:23.027665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.027631 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7"] Nov 01 22:42:23.034298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.034267 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af2843b-44fs7"] Nov 01 22:42:23.040137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.040110 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b"] Nov 01 22:42:23.045623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.045596 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr"] Nov 01 22:42:23.050428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.050408 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-381dcaf7-q2r6b"] Nov 01 22:42:23.055103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.055083 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06196609-9l9hr"] Nov 01 22:42:23.261741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.261703 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:23.450859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.450828 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 22:42:23.451047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:23.450891 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 22:42:24.087931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:24.087901 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:24.246711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:24.246681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:24.736219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:24.736184 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220066e3-6f27-44c8-bd95-843708b3095e" path="/var/lib/kubelet/pods/220066e3-6f27-44c8-bd95-843708b3095e/volumes" Nov 01 22:42:24.736696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:24.736680 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32810abe-3631-417c-89ba-da7a149a9abe" path="/var/lib/kubelet/pods/32810abe-3631-417c-89ba-da7a149a9abe/volumes" Nov 01 22:42:24.737101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:24.737088 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ed81d4-7c21-415f-b409-1aee30f3123e" path="/var/lib/kubelet/pods/63ed81d4-7c21-415f-b409-1aee30f3123e/volumes" Nov 01 22:42:24.981995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:24.981962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:25.724755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:25.724722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:26.596285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:26.596237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:26.988025 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:26.987994 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f"] Nov 01 22:42:26.994989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:26.994963 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:26.999814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:26.999790 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:42:26.999814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:26.999803 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:42:26.999992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:26.999846 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:42:26.999992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:26.999857 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:42:27.002442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.002421 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f"] Nov 01 22:42:27.121143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.121106 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46qj\" (UniqueName: \"kubernetes.io/projected/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-kube-api-access-r46qj\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.121354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.121335 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.121518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.121502 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.223619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.223558 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.223847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.223687 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r46qj\" (UniqueName: \"kubernetes.io/projected/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-kube-api-access-r46qj\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.223922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.223894 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.224681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.224648 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.226826 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.226795 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.232104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.232078 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46qj\" (UniqueName: \"kubernetes.io/projected/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-kube-api-access-r46qj\") pod \"maas-api-revoke-sub-bda0c090-mtm8f\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.334265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.334173 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:27.499096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.493611 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 22:42:27.499096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.494460 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 22:42:27.629024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.628941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:27.756730 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:42:27.756687 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434a2293_8f5a_4e20_8e20_7d3ca598ed8d.slice/crio-817b7b7c9ddb2a99dbb03e6dc04cfa0dced0aac13e2d4c9fa483809700263cdd WatchSource:0}: Error finding container 817b7b7c9ddb2a99dbb03e6dc04cfa0dced0aac13e2d4c9fa483809700263cdd: Status 404 returned error can't find the container with id 817b7b7c9ddb2a99dbb03e6dc04cfa0dced0aac13e2d4c9fa483809700263cdd Nov 01 22:42:27.758000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:27.757950 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f"] Nov 01 22:42:28.560117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:28.560071 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:28.651672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:28.651292 4981 generic.go:356] "Generic (PLEG): container finished" podID="434a2293-8f5a-4e20-8e20-7d3ca598ed8d" containerID="a10b9c8dbef0e49e9078e07395a3ed392861ca22fd5d014ed410e5b4f26436d7" exitCode=0 Nov 01 22:42:28.651672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:28.651415 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" event={"ID":"434a2293-8f5a-4e20-8e20-7d3ca598ed8d","Type":"ContainerDied","Data":"a10b9c8dbef0e49e9078e07395a3ed392861ca22fd5d014ed410e5b4f26436d7"} Nov 01 22:42:28.651672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:28.651490 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" event={"ID":"434a2293-8f5a-4e20-8e20-7d3ca598ed8d","Type":"ContainerStarted","Data":"817b7b7c9ddb2a99dbb03e6dc04cfa0dced0aac13e2d4c9fa483809700263cdd"} Nov 01 22:42:29.107781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.107745 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:29.240329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.240292 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:29.563230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.563195 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:29.854213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.854187 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:29.969089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.969050 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:29.996868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.996840 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-openshift-ca\") pod \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " Nov 01 22:42:29.997107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.997090 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-kube-api-access-r46qj\" (UniqueName: \"kubernetes.io/projected/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-kube-api-access-r46qj\") pod \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " Nov 01 22:42:29.997197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.997164 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-openshift-ca" pod "434a2293-8f5a-4e20-8e20-7d3ca598ed8d" (UID: "434a2293-8f5a-4e20-8e20-7d3ca598ed8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:42:29.997256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.997193 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-xks-ca\") pod \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\" (UID: \"434a2293-8f5a-4e20-8e20-7d3ca598ed8d\") " Nov 01 22:42:29.997984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.997957 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:29.999429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.999399 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-kube-api-access-r46qj" pod "434a2293-8f5a-4e20-8e20-7d3ca598ed8d" (UID: "434a2293-8f5a-4e20-8e20-7d3ca598ed8d"). InnerVolumeSpecName "kube-api-access-r46qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:42:29.999540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:29.999422 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-xks-ca" pod "434a2293-8f5a-4e20-8e20-7d3ca598ed8d" (UID: "434a2293-8f5a-4e20-8e20-7d3ca598ed8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:42:30.076623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.076515 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:30.099745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.099717 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46qj\" (UniqueName: \"kubernetes.io/projected/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-kube-api-access-r46qj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:30.099745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.099744 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/434a2293-8f5a-4e20-8e20-7d3ca598ed8d-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:42:30.210446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.208305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:30.495235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.495108 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:30.671682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.671650 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" Nov 01 22:42:30.671682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.671664 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f" event={"ID":"434a2293-8f5a-4e20-8e20-7d3ca598ed8d","Type":"ContainerDied","Data":"817b7b7c9ddb2a99dbb03e6dc04cfa0dced0aac13e2d4c9fa483809700263cdd"} Nov 01 22:42:30.672103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:30.671713 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="817b7b7c9ddb2a99dbb03e6dc04cfa0dced0aac13e2d4c9fa483809700263cdd" Nov 01 22:42:31.214427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:31.214392 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:32.205325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:32.205247 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:33.157343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:33.157315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:33.464598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:33.458507 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:33.852848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:33.852777 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:34.003172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.003132 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:34.494427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.494391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:34.656786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.656749 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j"] Nov 01 22:42:34.658029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.658004 4981 cpu_manager.go:414] "containerMap: removing container" podUID="434a2293-8f5a-4e20-8e20-7d3ca598ed8d" containerName="revoke-keys" Nov 01 22:42:34.658029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.658027 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434a2293-8f5a-4e20-8e20-7d3ca598ed8d" containerName="revoke-keys" Nov 01 22:42:34.658324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.658309 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="434a2293-8f5a-4e20-8e20-7d3ca598ed8d" containerName="revoke-keys" Nov 01 22:42:34.689597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.683841 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.689597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.689330 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j"] Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.739798 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.739856 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.739896 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.740012 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.740129 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.740172 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.740302 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.740331 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn7v8\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-kube-api-access-fn7v8\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.742632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.740432 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.814653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.805914 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-dc635e-openshift-default-dockercfg-fjhdk\"" Nov 01 22:42:34.814653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.807054 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:34.844005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.843968 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.844311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.844286 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.844522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.844503 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.844669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.844650 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7v8\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-kube-api-access-fn7v8\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.844884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.844866 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.845155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.845133 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.845365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.845346 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.845505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.845489 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.845793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.845774 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.846994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.846970 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istiod-ca-cert\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.847684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.847662 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-certs\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.848235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.848213 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-data\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.848686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.848645 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-credential-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.849115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.849094 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-socket\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.856841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.855863 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-podinfo\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.857765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.857739 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-envoy\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.882878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.882848 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7v8\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-kube-api-access-fn7v8\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:34.883026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:34.882955 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-token\") pod \"e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:35.082631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.082506 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:35.393397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.393323 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:35.455624 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:42:35.455051 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3934c5_9084_4799_b0fe_9b6ac6a054db.slice/crio-c8dc428cfe02124d1b5579733f47fd5545d795f5a2be3970325b208602e03152 WatchSource:0}: Error finding container c8dc428cfe02124d1b5579733f47fd5545d795f5a2be3970325b208602e03152: Status 404 returned error can't find the container with id c8dc428cfe02124d1b5579733f47fd5545d795f5a2be3970325b208602e03152 Nov 01 22:42:35.455624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.455516 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j"] Nov 01 22:42:35.459795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.459762 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:42:35.459916 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.459840 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:42:35.459916 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.459889 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:42:35.601598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.600093 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:35.810038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:35.810000 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" event={"ID":"4a3934c5-9084-4799-b0fe-9b6ac6a054db","Type":"ContainerStarted","Data":"c8dc428cfe02124d1b5579733f47fd5545d795f5a2be3970325b208602e03152"} Nov 01 22:42:36.259397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:36.259352 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:36.820665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:36.820626 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" event={"ID":"4a3934c5-9084-4799-b0fe-9b6ac6a054db","Type":"ContainerStarted","Data":"4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0"} Nov 01 22:42:36.855689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:36.855642 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" podStartSLOduration=2.855628295 podStartE2EDuration="2.855628295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:36.849605092 +0000 UTC m=+1122.277483024" watchObservedRunningTime="2026-11-01 22:42:36.855628295 +0000 UTC m=+1122.283506225" Nov 01 22:42:37.036679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.036628 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6"] Nov 01 22:42:37.042486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.042463 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aa639cd-g5gg6"] Nov 01 22:42:37.083822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.083724 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:37.089770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.089743 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:37.313311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.313275 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:37.420379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.420270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:37.830207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.830180 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:37.831196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:37.831177 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:42:38.403144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:38.403109 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:38.736109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:38.736075 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a81a22-143f-4faa-8ca5-b7d152c74c85" path="/var/lib/kubelet/pods/f8a81a22-143f-4faa-8ca5-b7d152c74c85/volumes" Nov 01 22:42:39.036433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:39.036345 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g"] Nov 01 22:42:39.042679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:39.042651 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f5fe5fe-hdk8g"] Nov 01 22:42:39.074473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:39.074448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:39.172868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:39.172836 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:39.391938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:39.391864 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:40.178682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.178638 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:40.315592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.313534 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:40.378707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.378677 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd"] Nov 01 22:42:40.384199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.384173 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.392307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.392272 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd"] Nov 01 22:42:40.394316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.394275 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-dc635e\"" Nov 01 22:42:40.394876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.394838 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-dc635e-dockercfg-5jfx5\"" Nov 01 22:42:40.411516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.411366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:40.414489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.414449 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5"] Nov 01 22:42:40.421461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.421436 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.431636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.431606 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5"] Nov 01 22:42:40.549660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.543148 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwn8\" (UniqueName: \"kubernetes.io/projected/b153d778-544a-450d-8518-f1536e77e787-kube-api-access-hvwn8\") pod \"payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.549660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.543406 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rv4\" (UniqueName: \"kubernetes.io/projected/41289d4e-dbd9-425d-8f5e-e46cd1baa791-kube-api-access-m4rv4\") pod \"payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.549660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.543470 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b153d778-544a-450d-8518-f1536e77e787-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.549660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.543633 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41289d4e-dbd9-425d-8f5e-e46cd1baa791-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.656628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.656539 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b153d778-544a-450d-8518-f1536e77e787-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.657881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.657856 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41289d4e-dbd9-425d-8f5e-e46cd1baa791-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.658715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.658696 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41289d4e-dbd9-425d-8f5e-e46cd1baa791-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.658838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.658763 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwn8\" (UniqueName: \"kubernetes.io/projected/b153d778-544a-450d-8518-f1536e77e787-kube-api-access-hvwn8\") pod \"payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.659079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.657634 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b153d778-544a-450d-8518-f1536e77e787-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.659289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.659270 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rv4\" (UniqueName: \"kubernetes.io/projected/41289d4e-dbd9-425d-8f5e-e46cd1baa791-kube-api-access-m4rv4\") pod \"payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.668234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.668178 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwn8\" (UniqueName: \"kubernetes.io/projected/b153d778-544a-450d-8518-f1536e77e787-kube-api-access-hvwn8\") pod \"payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.668332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.668178 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rv4\" (UniqueName: \"kubernetes.io/projected/41289d4e-dbd9-425d-8f5e-e46cd1baa791-kube-api-access-m4rv4\") pod \"payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.706110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.706017 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:40.738973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.738945 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:40.743285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:40.743254 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3d7c49-e789-40da-afc8-72f69100c0f6" path="/var/lib/kubelet/pods/1e3d7c49-e789-40da-afc8-72f69100c0f6/volumes" Nov 01 22:42:41.028142 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:42:41.028115 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41289d4e_dbd9_425d_8f5e_e46cd1baa791.slice/crio-8ad8753cf5979179ac6bf5003b4e656e76d6d0c338eb536d32d7510e7a098fa9 WatchSource:0}: Error finding container 8ad8753cf5979179ac6bf5003b4e656e76d6d0c338eb536d32d7510e7a098fa9: Status 404 returned error can't find the container with id 8ad8753cf5979179ac6bf5003b4e656e76d6d0c338eb536d32d7510e7a098fa9 Nov 01 22:42:41.030421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.030401 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd"] Nov 01 22:42:41.058053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.058024 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5"] Nov 01 22:42:41.228755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.228723 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:41.894262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.894221 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" event={"ID":"b153d778-544a-450d-8518-f1536e77e787","Type":"ContainerStarted","Data":"7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0"} Nov 01 22:42:41.894470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.894321 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" event={"ID":"b153d778-544a-450d-8518-f1536e77e787","Type":"ContainerStarted","Data":"56d7752d337c5884ddea2eb72680ac281ed746b4995161af8a9f94bca6bec208"} Nov 01 22:42:41.894470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.894352 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:41.896373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.896337 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" event={"ID":"41289d4e-dbd9-425d-8f5e-e46cd1baa791","Type":"ContainerStarted","Data":"8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34"} Nov 01 22:42:41.896513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.896395 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:41.896513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.896420 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" event={"ID":"41289d4e-dbd9-425d-8f5e-e46cd1baa791","Type":"ContainerStarted","Data":"8ad8753cf5979179ac6bf5003b4e656e76d6d0c338eb536d32d7510e7a098fa9"} Nov 01 22:42:41.913785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.913739 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" podStartSLOduration=1.913727601 podStartE2EDuration="1.913727601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:41.909222809 +0000 UTC m=+1127.337100741" watchObservedRunningTime="2026-11-01 22:42:41.913727601 +0000 UTC m=+1127.341605526" Nov 01 22:42:41.926132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.926091 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" podStartSLOduration=1.926080782 podStartE2EDuration="1.926080782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:41.924866213 +0000 UTC m=+1127.352744147" watchObservedRunningTime="2026-11-01 22:42:41.926080782 +0000 UTC m=+1127.353958712" Nov 01 22:42:41.942978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:41.942952 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:42.659793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:42.659764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:43.520788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:43.520757 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:43.619708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:43.619679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:44.497999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:44.497963 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:44.528608 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:42:44.528552 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:42:44.681093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:44.681061 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:45.465429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:45.465397 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:46.247821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:46.247779 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:46.886178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:46.886146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:47.038286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:47.038240 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g"] Nov 01 22:42:47.044742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:47.044716 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81701ab8-f7n5g"] Nov 01 22:42:47.682689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:47.682652 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:47.903389 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:47.903343 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:48.033225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:48.033147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:48.658398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:48.658366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:48.744012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:48.743974 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e6ea18-1cdb-4675-af33-90af7b44f788" path="/var/lib/kubelet/pods/44e6ea18-1cdb-4675-af33-90af7b44f788/volumes" Nov 01 22:42:49.401368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:49.401328 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:50.074005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:50.073964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:50.869745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:50.869716 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:51.592631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:51.592562 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:52.377753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:52.377713 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:52.910972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:52.910943 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 22:42:52.911377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:52.911022 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 22:42:53.085318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:53.085290 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:54.000929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:54.000898 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:55.016385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:55.016350 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:55.107969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:55.107935 4981 scope.go:122] "RemoveContainer" containerID="5dcb11bd76fadb9e92a44af60669c7fa01e5a93ae6bfa17cb68ab120a00ebc73" Nov 01 22:42:55.120739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:55.120720 4981 scope.go:122] "RemoveContainer" containerID="8ab69facae30df7528e8915647d81dd85515f7f3e06633dbaeb090691ef5c2ca" Nov 01 22:42:55.132268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:55.132246 4981 scope.go:122] "RemoveContainer" containerID="56c4d500ea0817b8364e7f3b151135fdfa4bad0805f9c88881de54f97bd9aab3" Nov 01 22:42:55.144269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:55.144244 4981 scope.go:122] "RemoveContainer" containerID="11eacb34a5fa9be2f9c3927755859478f00faf7de16bc8ad71f2848660f5d3dd" Nov 01 22:42:55.154770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:55.154753 4981 scope.go:122] "RemoveContainer" containerID="581e1ae6ac390811548547fb3641c0f2be9969f301f1152188a47622db34c311" Nov 01 22:42:55.645397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:55.645367 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:56.270884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:56.270852 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:56.974677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:56.974646 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:57.993741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:57.993692 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:58.670174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:58.670141 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:58.864812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:58.864780 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:59.028898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:59.028867 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc"] Nov 01 22:42:59.034092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:59.034069 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed54806f-wlnhc"] Nov 01 22:42:59.145803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:59.145775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:42:59.465704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:42:59.465679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:00.171065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:00.171036 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:00.736008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:00.735972 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aad3b0c-d4de-46f6-aa85-50f35e4907cc" path="/var/lib/kubelet/pods/8aad3b0c-d4de-46f6-aa85-50f35e4907cc/volumes" Nov 01 22:43:00.943162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:00.943126 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:01.046415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.046317 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:01.391100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.391022 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg"] Nov 01 22:43:01.398651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.398622 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.405017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.404975 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg"] Nov 01 22:43:01.406550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.406371 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-30578128-openshift-default-dockercfg-l7hql\"" Nov 01 22:43:01.434024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.433997 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:01.557598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.546174 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:01.557952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.556695 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-credential-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.558201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.558178 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.558384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.558364 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-podinfo\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.590596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.558606 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-data\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.591692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.591662 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-certs\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.592010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.591992 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-envoy\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.592184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.592171 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istiod-ca-cert\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.592427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.592412 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bqbr\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-kube-api-access-6bqbr\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.592615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.592600 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-token\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694261 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-token\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694387 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-credential-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694416 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694464 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-podinfo\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694522 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-data\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694562 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-certs\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694704 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-envoy\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694745 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istiod-ca-cert\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.694892 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bqbr\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-kube-api-access-6bqbr\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.695824 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-credential-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.696443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.696052 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-socket\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.709430 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.709395 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-token\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.710911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.709945 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-data\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.710911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.710868 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istiod-ca-cert\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.711346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.711310 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-certs\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.714849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.714826 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-envoy\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.724781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.716062 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-podinfo\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:01.734979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:01.733913 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bqbr\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-kube-api-access-6bqbr\") pod \"e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:02.016743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:02.016648 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:02.100781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:02.100695 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:02.321185 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:43:02.321152 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae1ca54_b47f_49eb_af0a_a2b5b65cb8f8.slice/crio-e9c087c84d699cbf3d4dfdead961890ca1410c3c1dc597b6b829c23f36211eba WatchSource:0}: Error finding container e9c087c84d699cbf3d4dfdead961890ca1410c3c1dc597b6b829c23f36211eba: Status 404 returned error can't find the container with id e9c087c84d699cbf3d4dfdead961890ca1410c3c1dc597b6b829c23f36211eba Nov 01 22:43:02.323197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:02.323136 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg"] Nov 01 22:43:02.324363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:02.324320 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:02.324547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:02.324527 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:02.324699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:02.324671 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:02.853155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:02.853121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:03.150333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:03.150253 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" event={"ID":"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8","Type":"ContainerStarted","Data":"d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768"} Nov 01 22:43:03.150333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:03.150319 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" event={"ID":"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8","Type":"ContainerStarted","Data":"e9c087c84d699cbf3d4dfdead961890ca1410c3c1dc597b6b829c23f36211eba"} Nov 01 22:43:03.172415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:03.172354 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podStartSLOduration=2.172335785 podStartE2EDuration="2.172335785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:03.169940932 +0000 UTC m=+1148.597818875" watchObservedRunningTime="2026-11-01 22:43:03.172335785 +0000 UTC m=+1148.600213715" Nov 01 22:43:03.872490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:03.872451 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:04.016991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:04.016958 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:04.024794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:04.023076 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:04.164472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:04.164370 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:04.165371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:04.165346 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:43:04.841636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:04.841566 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:05.486660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:05.486627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:05.588459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:05.588425 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:05.881177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:05.881090 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:06.950110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:06.950081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.019411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.019375 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z"] Nov 01 22:43:07.024257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.024240 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.046596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.038654 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-30578128\"" Nov 01 22:43:07.046596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.039100 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-30578128-dockercfg-bs8bk\"" Nov 01 22:43:07.046596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.039385 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z"] Nov 01 22:43:07.061631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.061600 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5"] Nov 01 22:43:07.067901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.067865 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.079046 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.078768 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5"] Nov 01 22:43:07.082429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.082400 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.097214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.097182 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa344f5f-11a6-481a-b8da-3b3a267953e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.097385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.097364 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfsg\" (UniqueName: \"kubernetes.io/projected/fa344f5f-11a6-481a-b8da-3b3a267953e6-kube-api-access-5hfsg\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.229649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.208703 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfsg\" (UniqueName: \"kubernetes.io/projected/fa344f5f-11a6-481a-b8da-3b3a267953e6-kube-api-access-5hfsg\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.229649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.208970 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51de728e-ffd0-4207-9f8b-ab64505e46bc-plugins-config-volume\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.229649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.209065 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6l4l\" (UniqueName: \"kubernetes.io/projected/51de728e-ffd0-4207-9f8b-ab64505e46bc-kube-api-access-r6l4l\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.229649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.209181 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa344f5f-11a6-481a-b8da-3b3a267953e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.229649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.210536 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa344f5f-11a6-481a-b8da-3b3a267953e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.239599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.233059 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfsg\" (UniqueName: \"kubernetes.io/projected/fa344f5f-11a6-481a-b8da-3b3a267953e6-kube-api-access-5hfsg\") pod \"payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.328596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.311050 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51de728e-ffd0-4207-9f8b-ab64505e46bc-plugins-config-volume\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.328596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.311167 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6l4l\" (UniqueName: \"kubernetes.io/projected/51de728e-ffd0-4207-9f8b-ab64505e46bc-kube-api-access-r6l4l\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.328596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.312489 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51de728e-ffd0-4207-9f8b-ab64505e46bc-plugins-config-volume\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.328596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.324907 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6l4l\" (UniqueName: \"kubernetes.io/projected/51de728e-ffd0-4207-9f8b-ab64505e46bc-kube-api-access-r6l4l\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.365602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.364927 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:07.391378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.387036 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:07.615369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.615208 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:07.807504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.795408 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z"] Nov 01 22:43:07.842651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:07.842415 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5"] Nov 01 22:43:08.364071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:08.363918 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" event={"ID":"51de728e-ffd0-4207-9f8b-ab64505e46bc","Type":"ContainerStarted","Data":"a35e8e7a80143c77a0450b95a7b9bf43985c006bc4144b84643d6e4ce02ba53d"} Nov 01 22:43:08.387090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:08.386995 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" event={"ID":"fa344f5f-11a6-481a-b8da-3b3a267953e6","Type":"ContainerStarted","Data":"dddd24760bc54d4b55ff8fa6e473fc06cc1c2da5b0f81d1d9f0371af78729e79"} Nov 01 22:43:08.891601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:08.886502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:09.440627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:09.440561 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" event={"ID":"51de728e-ffd0-4207-9f8b-ab64505e46bc","Type":"ContainerStarted","Data":"3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634"} Nov 01 22:43:09.441691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:09.441668 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:09.444328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:09.444304 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" event={"ID":"fa344f5f-11a6-481a-b8da-3b3a267953e6","Type":"ContainerStarted","Data":"32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd"} Nov 01 22:43:09.445310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:09.445287 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:09.470596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:09.467399 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" podStartSLOduration=2.467383169 podStartE2EDuration="2.467383169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:09.464698869 +0000 UTC m=+1154.892576800" watchObservedRunningTime="2026-11-01 22:43:09.467383169 +0000 UTC m=+1154.895261098" Nov 01 22:43:09.517168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:09.517102 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" podStartSLOduration=3.517082334 podStartE2EDuration="3.517082334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:09.486291421 +0000 UTC m=+1154.914169354" watchObservedRunningTime="2026-11-01 22:43:09.517082334 +0000 UTC m=+1154.944960268" Nov 01 22:43:09.958739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:09.958663 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:10.621906 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.621859 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:10.640872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.640837 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:10.694796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.694753 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:43:10.697678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.697187 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:10.755919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.755264 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:10.756858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.756827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:10.892776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.892666 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:10.893044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.893010 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:10.975621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.957690 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:43:10.975621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.957768 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:10.978185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.978143 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:43:10.990196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:10.990162 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:11.160799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.160671 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.256597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.256053 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.264038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.257842 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:43:11.329232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.329193 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:43:11.340525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.337705 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.340525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.337761 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.340525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.337783 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:43:11.340525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.337804 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:43:11.365693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.365662 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:11.370809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.370772 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.370983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.370791 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.627510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.627469 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.628592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.628552 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:43:11.717598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.717174 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:11.717598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.717556 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:43:11.742599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.718133 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:43:11.742599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:11.718190 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:43:12.205333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.173991 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:43:12.205333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.175812 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:43:12.205333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.175866 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:43:12.205333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.191736 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.205333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.192182 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:43:12.239458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.239427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:12.258002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.256406 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.283628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.258953 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:43:12.319627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.315691 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.319627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.316642 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:43:12.362763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.360167 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.362763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.360868 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:43:12.431676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.419454 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:43:12.440595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.432063 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.440595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.432396 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:43:12.440595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.432935 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:43:12.456015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.441823 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:43:12.456015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.441893 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:43:12.456015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.441945 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:43:12.456015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.444394 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:12.485353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.470734 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:43:12.673605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.609523 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:12.674228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.632401 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:43:12.674454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.674436 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:43:12.674571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.674562 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:43:12.698605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.697150 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:43:12.733399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.733305 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:43:12.782048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.781889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:12.807401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.807369 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:43:12.807652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.807425 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:43:12.807652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.807456 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:43:12.807652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.807482 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:43:12.807652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:12.807510 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:43:13.429188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.429084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:13.592077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.592046 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj"] Nov 01 22:43:13.606419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.606388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:13.611440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.608232 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.614620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.614554 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj"] Nov 01 22:43:13.617662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.616811 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:43:13.617662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.616963 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:43:13.619567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.619538 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:43:13.619567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.619635 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:43:13.659918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.659884 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.660146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.660125 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.660345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.660330 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwsm\" (UniqueName: \"kubernetes.io/projected/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-kube-api-access-zxwsm\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.767216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.761881 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.767216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.762002 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwsm\" (UniqueName: \"kubernetes.io/projected/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-kube-api-access-zxwsm\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.767216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.762430 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.767216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.763233 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.767216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.765150 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.817991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.817956 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwsm\" (UniqueName: \"kubernetes.io/projected/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-kube-api-access-zxwsm\") pod \"maas-api-revoke-sub-fbf233fc-b47tj\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:13.929695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:13.929662 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:14.241996 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:43:14.241959 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2fa6c6_4d86_418d_a5ce_ac6565a6820c.slice/crio-fcf653567fafbe92257536c06b94a177d6d7599a72b735108ba738e3fafb7eda WatchSource:0}: Error finding container fcf653567fafbe92257536c06b94a177d6d7599a72b735108ba738e3fafb7eda: Status 404 returned error can't find the container with id fcf653567fafbe92257536c06b94a177d6d7599a72b735108ba738e3fafb7eda Nov 01 22:43:14.243746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:14.243721 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj"] Nov 01 22:43:14.370820 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:14.370780 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:14.538222 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:43:14.538112 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:43:14.841191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:14.841095 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" event={"ID":"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c","Type":"ContainerStarted","Data":"a18c6eb194dda33227d051f65ad778e86859734fa26afcf4ff5a8abc2677c577"} Nov 01 22:43:14.841191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:14.841154 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" event={"ID":"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c","Type":"ContainerStarted","Data":"fcf653567fafbe92257536c06b94a177d6d7599a72b735108ba738e3fafb7eda"} Nov 01 22:43:14.882194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:14.882147 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" podStartSLOduration=1.8821335590000001 podStartE2EDuration="1.882133559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:14.875083639 +0000 UTC m=+1160.302961566" watchObservedRunningTime="2026-11-01 22:43:14.882133559 +0000 UTC m=+1160.310011522" Nov 01 22:43:15.080665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:15.080630 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:15.769714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:15.769680 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:15.864484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:15.864445 4981 generic.go:356] "Generic (PLEG): container finished" podID="9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" containerID="a18c6eb194dda33227d051f65ad778e86859734fa26afcf4ff5a8abc2677c577" exitCode=0 Nov 01 22:43:15.864927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:15.864702 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" event={"ID":"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c","Type":"ContainerDied","Data":"a18c6eb194dda33227d051f65ad778e86859734fa26afcf4ff5a8abc2677c577"} Nov 01 22:43:16.789302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:16.789269 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:17.064335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.064273 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:17.119439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.119401 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-openshift-ca\") pod \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " Nov 01 22:43:17.119688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.119527 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-xks-ca\") pod \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " Nov 01 22:43:17.119796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.119702 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-kube-api-access-zxwsm\" (UniqueName: \"kubernetes.io/projected/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-kube-api-access-zxwsm\") pod \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\" (UID: \"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c\") " Nov 01 22:43:17.120288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.120250 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-openshift-ca" pod "9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" (UID: "9d2fa6c6-4d86-418d-a5ce-ac6565a6820c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:43:17.120832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.120805 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:17.122564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.122537 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-kube-api-access-zxwsm" pod "9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" (UID: "9d2fa6c6-4d86-418d-a5ce-ac6565a6820c"). InnerVolumeSpecName "kube-api-access-zxwsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:43:17.123936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.123018 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-xks-ca" pod "9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" (UID: "9d2fa6c6-4d86-418d-a5ce-ac6565a6820c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:43:17.226766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.226726 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:17.226766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.226772 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwsm\" (UniqueName: \"kubernetes.io/projected/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c-kube-api-access-zxwsm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:43:17.601039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.600996 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:17.719665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.716381 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:17.889149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.889076 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" Nov 01 22:43:17.889149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.889108 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj" event={"ID":"9d2fa6c6-4d86-418d-a5ce-ac6565a6820c","Type":"ContainerDied","Data":"fcf653567fafbe92257536c06b94a177d6d7599a72b735108ba738e3fafb7eda"} Nov 01 22:43:17.889149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:17.889151 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcf653567fafbe92257536c06b94a177d6d7599a72b735108ba738e3fafb7eda" Nov 01 22:43:18.028183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.028151 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf"] Nov 01 22:43:18.033699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.033669 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87a7b34d-bsxsf"] Nov 01 22:43:18.635698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.635571 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:18.750271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.750239 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b2dd81-71a6-4bc5-88b7-bc969373fe50" path="/var/lib/kubelet/pods/13b2dd81-71a6-4bc5-88b7-bc969373fe50/volumes" Nov 01 22:43:18.899816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.899735 4981 generic.go:356] "Generic (PLEG): container finished" podID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerID="4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0" exitCode=0 Nov 01 22:43:18.899966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.899811 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" event={"ID":"4a3934c5-9084-4799-b0fe-9b6ac6a054db","Type":"ContainerDied","Data":"4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0"} Nov 01 22:43:18.900265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.900251 4981 scope.go:122] "RemoveContainer" containerID="4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0" Nov 01 22:43:18.901942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.901916 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:18.902031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.901980 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:18.902031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:18.902011 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:19.434541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:19.434501 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:19.912635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:19.912533 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" event={"ID":"4a3934c5-9084-4799-b0fe-9b6ac6a054db","Type":"ContainerStarted","Data":"b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f"} Nov 01 22:43:20.093598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:20.083164 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:43:20.099604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:20.098841 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:43:20.099604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:20.098901 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 22:43:20.297599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:20.297006 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:21.098643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:21.097623 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:21.099164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:21.099017 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:43:21.099164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:21.099099 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 22:43:21.700635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:21.700029 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:43:21.719104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:21.719082 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:43:21.766772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:21.766738 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:22.027058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:22.026973 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5"] Nov 01 22:43:22.032427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:22.032401 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fc0dd06-q5zb5"] Nov 01 22:43:22.095599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:22.086952 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:43:22.095599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:22.087008 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 22:43:22.416375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:22.416278 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:22.749939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:22.749905 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58f2718-f203-44c9-b9b2-514c15ed14cb" path="/var/lib/kubelet/pods/f58f2718-f203-44c9-b9b2-514c15ed14cb/volumes" Nov 01 22:43:22.830337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:22.830307 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:43:23.098655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:23.097618 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:43:23.098655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:23.097677 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 22:43:23.204881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:23.204849 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:23.816934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:23.816901 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:23.912788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:23.912755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:24.087656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:24.082975 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 01 22:43:24.087656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:24.083035 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 01 22:43:24.565501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:24.565473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:25.087410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:25.087381 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:43:25.088288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:25.088268 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 22:43:25.594287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:25.594251 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:26.446450 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:26.446419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:26.964995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:26.964962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:27.082115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:27.082085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:27.465743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:27.465690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:27.610602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:27.608015 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:28.232340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:28.232310 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:29.478376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:29.478344 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:30.270561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:30.270535 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:30.517884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:30.517854 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:31.267620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:31.267536 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:31.773508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:31.773476 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:32.273943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:32.273901 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:32.526656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:32.526540 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:32.923266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:32.923231 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:33.064732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:33.064699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:33.546613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:33.544261 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:34.321698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.321669 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:34.423346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.423310 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92"] Nov 01 22:43:34.424772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.424714 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" containerName="revoke-keys" Nov 01 22:43:34.424772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.424740 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" containerName="revoke-keys" Nov 01 22:43:34.424974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.424958 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" containerName="revoke-keys" Nov 01 22:43:34.430994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.430976 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:34.452461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.452414 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4534ad\"" Nov 01 22:43:34.456917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.453536 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4534ad-dockercfg-l56mq\"" Nov 01 22:43:34.464318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.463871 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92"] Nov 01 22:43:34.469506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.469404 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72fb\" (UniqueName: \"kubernetes.io/projected/1609f345-e130-47a0-a92d-e7ae55616848-kube-api-access-b72fb\") pod \"payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:34.469989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.469890 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1609f345-e130-47a0-a92d-e7ae55616848-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:34.569601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.555264 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:34.868830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.571827 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72fb\" (UniqueName: \"kubernetes.io/projected/1609f345-e130-47a0-a92d-e7ae55616848-kube-api-access-b72fb\") pod \"payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:34.869261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.869235 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1609f345-e130-47a0-a92d-e7ae55616848-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:34.891647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.870754 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1609f345-e130-47a0-a92d-e7ae55616848-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:34.891647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:34.891045 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72fb\" (UniqueName: \"kubernetes.io/projected/1609f345-e130-47a0-a92d-e7ae55616848-kube-api-access-b72fb\") pod \"payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:35.070915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:35.070877 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:35.321571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:35.321485 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:35.419919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:35.419891 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92"] Nov 01 22:43:36.183199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:36.183161 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" event={"ID":"1609f345-e130-47a0-a92d-e7ae55616848","Type":"ContainerStarted","Data":"7ea15d31f267f39706d88cce66e2fa18fd4d6015ce551ba29180c30407f37fe5"} Nov 01 22:43:36.183449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:36.183427 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" event={"ID":"1609f345-e130-47a0-a92d-e7ae55616848","Type":"ContainerStarted","Data":"be1ec5f7e5487ac2f6ef743768adacb96cceb727cd7b270e01062dba5cf514a2"} Nov 01 22:43:36.183634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:36.183617 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:36.946598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:36.939846 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:38.334553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:38.333876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:38.394122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:38.392691 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:38.447665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:38.446690 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:38.648994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:38.648892 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:43:39.273052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:39.273021 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:40.116081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:40.116035 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:40.995005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:40.994972 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:41.781731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:41.781704 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:42.023933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:42.023878 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" podStartSLOduration=8.023861249 podStartE2EDuration="8.023861249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:36.211935226 +0000 UTC m=+1181.639813156" watchObservedRunningTime="2026-11-01 22:43:42.023861249 +0000 UTC m=+1187.451739239" Nov 01 22:43:42.029027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:42.029002 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml"] Nov 01 22:43:42.034629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:42.034550 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-289483c7-khlml"] Nov 01 22:43:42.463289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:42.463243 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:42.737121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:42.737041 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f00b8dd-9abd-4b90-9383-bbd52bb4bc88" path="/var/lib/kubelet/pods/7f00b8dd-9abd-4b90-9383-bbd52bb4bc88/volumes" Nov 01 22:43:43.172149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:43.172065 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:43.879882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:43.879834 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:44.526383 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:43:44.526344 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:43:44.567565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:44.567535 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:45.420723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:45.420694 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:45.535647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:45.535518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:46.453358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:46.453323 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:47.167438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:47.167408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:47.248743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:47.240288 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:43:48.060471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:48.060428 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:48.761460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:48.761428 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:49.490354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:49.490304 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:50.213055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:50.213025 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:50.874052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:50.874019 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:51.031112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:51.031080 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2"] Nov 01 22:43:51.037188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:51.037162 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4150dc57-twhd2"] Nov 01 22:43:51.658780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:51.658752 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:52.329880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:52.329848 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:52.737112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:52.737082 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8" path="/var/lib/kubelet/pods/c20d2e0c-8a55-42f5-adf8-71d8bb3a8fc8/volumes" Nov 01 22:43:53.123288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:53.123193 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:54.083925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:54.083879 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:54.764809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:54.764774 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:55.233274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.233236 4981 scope.go:122] "RemoveContainer" containerID="5406fbaba17b54bae1e6deb8ea0abe696a1e06e269266d1761cabc571369b310" Nov 01 22:43:55.250676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.250654 4981 scope.go:122] "RemoveContainer" containerID="c2313e74d0c10b68e34a2c8d7407debb0097c30e988da22fbdbbf2b1c1c65f8c" Nov 01 22:43:55.262669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.262646 4981 scope.go:122] "RemoveContainer" containerID="1931426763ac9c25cf1f1132cb3dfbc26b2f4a4d3808fa132d5501b8b16d9732" Nov 01 22:43:55.275126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.275104 4981 scope.go:122] "RemoveContainer" containerID="414db64fc857aeb2e8800c92cd899e5c3361116c53bacc7d9813639f7e3d6c88" Nov 01 22:43:55.285458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.285441 4981 scope.go:122] "RemoveContainer" containerID="3d07c46586a3a7ea2c21dac4ae12914a5a880ea2ea22c98033aec362bb43f74a" Nov 01 22:43:55.294888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.294868 4981 scope.go:122] "RemoveContainer" containerID="2a6ecef2d99625b1526f5025f039dae3db86f0aad3a9da331587d67da895bdf2" Nov 01 22:43:55.304440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.304424 4981 scope.go:122] "RemoveContainer" containerID="6f3ca318211b0807c27f834daaef99cd0a663c303f3ebe089c19575598169697" Nov 01 22:43:55.314838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.314812 4981 scope.go:122] "RemoveContainer" containerID="6e3afb8e84ee3fd78b928108fe2dcd641f63b05608cbdb89e678775f8f39129d" Nov 01 22:43:55.324478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.324454 4981 scope.go:122] "RemoveContainer" containerID="1121fac32c5a4821630fd8a9975e714e1de30b238a9c3c4e1c4d6b0a9c19f218" Nov 01 22:43:55.334134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.334113 4981 scope.go:122] "RemoveContainer" containerID="23c84757269d8fc3a8c5ef60acce0d5d1ae3a6c48a77b314a0ebd222a41b7811" Nov 01 22:43:55.343861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.343842 4981 scope.go:122] "RemoveContainer" containerID="dcf2d08e0955cfb2ae40d4a4d402363f05e00c458a8d7ceb8e1a1b0eb7c484d8" Nov 01 22:43:55.353798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.353782 4981 scope.go:122] "RemoveContainer" containerID="7c93948d0717eb357dfb3d0b39a87cf8cb45261779fb13c7b0136fce91d4d8be" Nov 01 22:43:55.364185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.364169 4981 scope.go:122] "RemoveContainer" containerID="b2757e16babd7d4f01e96d6fcc47e28c98ecc3be3fe071230714ba945b6ff9bb" Nov 01 22:43:55.374790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.374773 4981 scope.go:122] "RemoveContainer" containerID="c6136a0262f640860d237906f1e5dff993ef3610c7ed52c6be47795e2532e19c" Nov 01 22:43:55.385461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.385442 4981 scope.go:122] "RemoveContainer" containerID="3c980dca3f7b462471dbcc437685b0d4782316272f68b9438aa38a697df5e1a5" Nov 01 22:43:55.514461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:55.514434 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:56.220730 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:56.220690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:57.032017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:57.031980 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh"] Nov 01 22:43:57.044975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:57.044937 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610d2ea-8kddh"] Nov 01 22:43:57.270399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:57.270364 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:57.948636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:57.948563 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:58.186312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.186279 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:58.525296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.525260 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:58.738029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.737988 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0940e4-3d14-4617-a94e-53e71ec20ff6" path="/var/lib/kubelet/pods/ea0940e4-3d14-4617-a94e-53e71ec20ff6/volumes" Nov 01 22:43:58.925498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.925466 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:58.929173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.929143 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd"] Nov 01 22:43:58.938560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.938539 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.949818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.949787 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd"] Nov 01 22:43:58.973591 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.973543 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-envoy\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.973861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.973839 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml298\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-kube-api-access-ml298\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.973971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.973950 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-credential-socket\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.974061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.974042 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-socket\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.974136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.974120 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-data\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.974195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.974151 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istiod-ca-cert\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.974255 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.974201 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-podinfo\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.974354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.974339 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-certs\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.974413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.974375 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-token\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:58.981521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:58.981501 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-df4b43-openshift-default-dockercfg-zs4js\"" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102261 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-certs\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102326 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-token\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102405 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-envoy\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102683 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml298\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-kube-api-access-ml298\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102786 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-credential-socket\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102875 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-socket\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102907 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-data\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102937 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istiod-ca-cert\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.106600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.102992 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-podinfo\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.117604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.110310 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-certs\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.121604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.118124 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:59.165633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.165341 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-credential-socket\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.165819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.165646 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-socket\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.171630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.165925 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-data\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.171630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.166604 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istiod-ca-cert\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.180923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.180743 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml298\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-kube-api-access-ml298\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.198597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.185138 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-token\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.198597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.189347 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-podinfo\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.207598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.203150 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9"] Nov 01 22:43:59.207598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.206870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-envoy\") pod \"e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.220849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.213899 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d605a0e0-kths9"] Nov 01 22:43:59.238435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.238394 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:43:59.259417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.259371 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:43:59.562784 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:43:59.562754 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bef9cd2_8fd6_42e4_865c_0004f50dd88a.slice/crio-b85412c149c75f2730c419833d5c06e0d044d3c045bdf74b2a77cb3e3953ab33 WatchSource:0}: Error finding container b85412c149c75f2730c419833d5c06e0d044d3c045bdf74b2a77cb3e3953ab33: Status 404 returned error can't find the container with id b85412c149c75f2730c419833d5c06e0d044d3c045bdf74b2a77cb3e3953ab33 Nov 01 22:43:59.564080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.564058 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd"] Nov 01 22:43:59.565658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.565630 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:59.565733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.565695 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:59.565733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.565724 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:43:59.788306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:43:59.788268 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:00.029590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.029546 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw"] Nov 01 22:44:00.035180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.035145 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d4a5c06-zbzcw"] Nov 01 22:44:00.491049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.491017 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:00.516623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.516557 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" event={"ID":"1bef9cd2-8fd6-42e4-865c-0004f50dd88a","Type":"ContainerStarted","Data":"10f7f23bf0a0ecbc00644104539551629839257ee93574c9504540714f59cb7f"} Nov 01 22:44:00.516823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.516688 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" event={"ID":"1bef9cd2-8fd6-42e4-865c-0004f50dd88a","Type":"ContainerStarted","Data":"b85412c149c75f2730c419833d5c06e0d044d3c045bdf74b2a77cb3e3953ab33"} Nov 01 22:44:00.576650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.576589 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podStartSLOduration=2.5765553949999997 podStartE2EDuration="2.576555395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:00.550403705 +0000 UTC m=+1205.978281638" watchObservedRunningTime="2026-11-01 22:44:00.576555395 +0000 UTC m=+1206.004433326" Nov 01 22:44:00.774019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.773925 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818e5671-c308-4e66-ad34-5b92925dee1c" path="/var/lib/kubelet/pods/818e5671-c308-4e66-ad34-5b92925dee1c/volumes" Nov 01 22:44:00.774643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:00.774620 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59b9d86-b77c-4ffd-8926-04043572f32a" path="/var/lib/kubelet/pods/b59b9d86-b77c-4ffd-8926-04043572f32a/volumes" Nov 01 22:44:01.260053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.260017 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:44:01.265355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.265326 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:01.266104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.266084 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:44:01.379338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.379304 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd"] Nov 01 22:44:01.386175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.386153 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.400066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.400036 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:01.400225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.400174 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:44:01.406913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.406887 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd"] Nov 01 22:44:01.407643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.407566 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:44:01.407759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.407646 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:44:01.449395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.446149 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vhx\" (UniqueName: \"kubernetes.io/projected/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-kube-api-access-c8vhx\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.449395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.446244 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.449395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.446631 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.526728 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.526642 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:44:01.527403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.527388 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:44:01.547932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.547904 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vhx\" (UniqueName: \"kubernetes.io/projected/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-kube-api-access-c8vhx\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.548073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.547970 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.548276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.548260 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.548954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.548933 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.550453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.550428 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.579856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.579827 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vhx\" (UniqueName: \"kubernetes.io/projected/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-kube-api-access-c8vhx\") pod \"maas-api-revoke-sub-183e4b75-snrdd\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:01.711633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:01.711522 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:02.009564 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:44:02.009507 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98e2cd5_dbb7_4fdf_ab08_7ed8d5178f3b.slice/crio-bfce0b50a6620c12622dbe3bd2101c0719c4a5a79a6598a4c889f92b5608bd37 WatchSource:0}: Error finding container bfce0b50a6620c12622dbe3bd2101c0719c4a5a79a6598a4c889f92b5608bd37: Status 404 returned error can't find the container with id bfce0b50a6620c12622dbe3bd2101c0719c4a5a79a6598a4c889f92b5608bd37 Nov 01 22:44:02.010977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:02.010943 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd"] Nov 01 22:44:02.277274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:02.277237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:02.538693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:02.538570 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" event={"ID":"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b","Type":"ContainerStarted","Data":"59e330a30be34e929f9da551f277acdccb1ae111e40bc3f1ac981c855b97ae52"} Nov 01 22:44:02.539153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:02.538716 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" event={"ID":"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b","Type":"ContainerStarted","Data":"bfce0b50a6620c12622dbe3bd2101c0719c4a5a79a6598a4c889f92b5608bd37"} Nov 01 22:44:02.559760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:02.559703 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" podStartSLOduration=1.559689132 podStartE2EDuration="1.559689132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:02.556035547 +0000 UTC m=+1207.983913481" watchObservedRunningTime="2026-11-01 22:44:02.559689132 +0000 UTC m=+1207.987567063" Nov 01 22:44:02.936503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:02.936463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:03.050809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:03.050765 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp"] Nov 01 22:44:03.060379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:03.060342 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ef5042-qwcdp"] Nov 01 22:44:03.082894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:03.082857 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:03.223654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:03.223436 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:03.550526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:03.550446 4981 generic.go:356] "Generic (PLEG): container finished" podID="f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" containerID="59e330a30be34e929f9da551f277acdccb1ae111e40bc3f1ac981c855b97ae52" exitCode=0 Nov 01 22:44:03.550918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:03.550534 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" event={"ID":"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b","Type":"ContainerDied","Data":"59e330a30be34e929f9da551f277acdccb1ae111e40bc3f1ac981c855b97ae52"} Nov 01 22:44:03.983241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:03.983209 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:04.218772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.218727 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:04.276094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.276010 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl"] Nov 01 22:44:04.281528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.281505 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:04.291700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.290617 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-df4b43\"" Nov 01 22:44:04.291700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.290847 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-df4b43-dockercfg-v2g2j\"" Nov 01 22:44:04.298309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.298259 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl"] Nov 01 22:44:04.342430 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.342401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:04.492599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.491796 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:04.492599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.492075 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvn4d\" (UniqueName: \"kubernetes.io/projected/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-kube-api-access-zvn4d\") pod \"payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:04.608172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.606367 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvn4d\" (UniqueName: \"kubernetes.io/projected/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-kube-api-access-zvn4d\") pod \"payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:04.610569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.608924 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:04.610569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.609949 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-plugins-config-volume\") pod \"payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:04.643035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.643001 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvn4d\" (UniqueName: \"kubernetes.io/projected/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-kube-api-access-zvn4d\") pod \"payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:04.753858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.753281 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29207512-8f8c-4067-849a-6fd359b2185a" path="/var/lib/kubelet/pods/29207512-8f8c-4067-849a-6fd359b2185a/volumes" Nov 01 22:44:04.856880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.856816 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:04.913695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.913642 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-xks-ca\") pod \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " Nov 01 22:44:04.913864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.913757 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-openshift-ca\") pod \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " Nov 01 22:44:04.913932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.913879 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-kube-api-access-c8vhx\" (UniqueName: \"kubernetes.io/projected/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-kube-api-access-c8vhx\") pod \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\" (UID: \"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b\") " Nov 01 22:44:04.914132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.914100 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-openshift-ca" pod "f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" (UID: "f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:04.914715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.914690 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:04.915994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.915962 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-xks-ca" pod "f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" (UID: "f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:04.916094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.916016 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-kube-api-access-c8vhx" pod "f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" (UID: "f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b"). InnerVolumeSpecName "kube-api-access-c8vhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:04.916094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:04.916050 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:05.017083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.017042 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:05.017083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.017085 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vhx\" (UniqueName: \"kubernetes.io/projected/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b-kube-api-access-c8vhx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:05.058847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.058759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:05.222072 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:44:05.222040 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f42c4d_cebd_462d_a59c_4252fa0b8e78.slice/crio-53bf4a29231965f30473f100230adf9f4aba9ffd22dce0fdd96cae18b2df39dc WatchSource:0}: Error finding container 53bf4a29231965f30473f100230adf9f4aba9ffd22dce0fdd96cae18b2df39dc: Status 404 returned error can't find the container with id 53bf4a29231965f30473f100230adf9f4aba9ffd22dce0fdd96cae18b2df39dc Nov 01 22:44:05.223120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.223096 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl"] Nov 01 22:44:05.595540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.595459 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" Nov 01 22:44:05.595540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.595490 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd" event={"ID":"f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b","Type":"ContainerDied","Data":"bfce0b50a6620c12622dbe3bd2101c0719c4a5a79a6598a4c889f92b5608bd37"} Nov 01 22:44:05.595812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.595546 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfce0b50a6620c12622dbe3bd2101c0719c4a5a79a6598a4c889f92b5608bd37" Nov 01 22:44:05.598970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.598939 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:05.599134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.599039 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" event={"ID":"f3f42c4d-cebd-462d-a59c-4252fa0b8e78","Type":"ContainerStarted","Data":"21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33"} Nov 01 22:44:05.599134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.599097 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" event={"ID":"f3f42c4d-cebd-462d-a59c-4252fa0b8e78","Type":"ContainerStarted","Data":"53bf4a29231965f30473f100230adf9f4aba9ffd22dce0fdd96cae18b2df39dc"} Nov 01 22:44:05.599270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.599144 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:05.641425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:05.641375 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" podStartSLOduration=1.641361925 podStartE2EDuration="1.641361925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:05.635650011 +0000 UTC m=+1211.063527943" watchObservedRunningTime="2026-11-01 22:44:05.641361925 +0000 UTC m=+1211.069239891" Nov 01 22:44:06.072391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:06.072345 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:06.204462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:06.204426 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:06.521322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:06.521290 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:06.922200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:06.922113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:07.821357 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:07.821320 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:07.918133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:07.918084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:08.176557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:08.176214 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:08.304608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:08.303053 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:09.355351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:09.355313 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:09.851505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:09.851466 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:09.949981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:09.949938 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:10.764492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:10.764460 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:11.199203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.199162 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn"] Nov 01 22:44:11.200047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.200029 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" containerName="revoke-keys" Nov 01 22:44:11.200047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.200047 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" containerName="revoke-keys" Nov 01 22:44:11.200220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.200211 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" containerName="revoke-keys" Nov 01 22:44:11.204657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.204639 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:11.211895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.211870 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a9f94e-dockercfg-h8hz6\"" Nov 01 22:44:11.212084 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.212062 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a9f94e\"" Nov 01 22:44:11.219393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.219372 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn"] Nov 01 22:44:11.249420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.249376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:11.390405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.379298 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:11.417924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.410086 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdnr\" (UniqueName: \"kubernetes.io/projected/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-kube-api-access-zjdnr\") pod \"payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:11.417924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.410614 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:11.526678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.524269 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:11.526678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.524373 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdnr\" (UniqueName: \"kubernetes.io/projected/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-kube-api-access-zjdnr\") pod \"payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:11.526678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.526160 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:11.592502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.592400 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdnr\" (UniqueName: \"kubernetes.io/projected/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-kube-api-access-zjdnr\") pod \"payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:11.823536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:11.823442 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:12.034911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.034879 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:12.114029 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:44:12.113999 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6ddf6f_99e9_4569_ba8a_f79b1dfef438.slice/crio-fba2d6c4496ce7f4b08bd8693ad9df66c5e25ab7f84ce12fbdb2673bad6c2b3f WatchSource:0}: Error finding container fba2d6c4496ce7f4b08bd8693ad9df66c5e25ab7f84ce12fbdb2673bad6c2b3f: Status 404 returned error can't find the container with id fba2d6c4496ce7f4b08bd8693ad9df66c5e25ab7f84ce12fbdb2673bad6c2b3f Nov 01 22:44:12.115164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.115136 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn"] Nov 01 22:44:12.269122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.269088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:12.546339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.546307 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:12.683320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.683279 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" event={"ID":"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438","Type":"ContainerStarted","Data":"f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac"} Nov 01 22:44:12.683529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.683343 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" event={"ID":"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438","Type":"ContainerStarted","Data":"fba2d6c4496ce7f4b08bd8693ad9df66c5e25ab7f84ce12fbdb2673bad6c2b3f"} Nov 01 22:44:12.683529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.683383 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:12.712117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.712065 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" podStartSLOduration=1.712051166 podStartE2EDuration="1.712051166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:12.709591546 +0000 UTC m=+1218.137469480" watchObservedRunningTime="2026-11-01 22:44:12.712051166 +0000 UTC m=+1218.139929096" Nov 01 22:44:12.721414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:12.721349 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:13.671873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:13.671843 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:13.978426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:13.978389 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:14.250511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.250398 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:14.333318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.333275 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6"] Nov 01 22:44:14.338563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.338536 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:14.357872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.357845 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6"] Nov 01 22:44:14.443299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.443264 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2a5c3234-dockercfg-jfkcl\"" Nov 01 22:44:14.557711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.457788 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2a5c3234\"" Nov 01 22:44:14.558164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.491950 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:14.601834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.601789 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cda48-24e5-4231-9e7a-67866b18b48b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:14.602219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.602196 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbwf\" (UniqueName: \"kubernetes.io/projected/9f2cda48-24e5-4231-9e7a-67866b18b48b-kube-api-access-hdbwf\") pod \"payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:14.646008 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:44:14.645960 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:44:14.704153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.704116 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbwf\" (UniqueName: \"kubernetes.io/projected/9f2cda48-24e5-4231-9e7a-67866b18b48b-kube-api-access-hdbwf\") pod \"payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:14.704589 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.704549 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cda48-24e5-4231-9e7a-67866b18b48b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:14.705459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.705436 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cda48-24e5-4231-9e7a-67866b18b48b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:14.713141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.713117 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbwf\" (UniqueName: \"kubernetes.io/projected/9f2cda48-24e5-4231-9e7a-67866b18b48b-kube-api-access-hdbwf\") pod \"payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:14.969135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:14.967917 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:15.442482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:15.442375 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6"] Nov 01 22:44:15.800599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:15.800279 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" event={"ID":"9f2cda48-24e5-4231-9e7a-67866b18b48b","Type":"ContainerStarted","Data":"879c32e6ce912df119aa3d453c484d364bf8f85b56604da2bce28ec43620c5b2"} Nov 01 22:44:16.415164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:16.415092 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:16.640131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:16.638867 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:44:17.096673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:17.093871 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" event={"ID":"9f2cda48-24e5-4231-9e7a-67866b18b48b","Type":"ContainerStarted","Data":"eef8693a0fcd1fab0533b3b08f4cd4c23afec5fc103afad53e65ffc261c1e925"} Nov 01 22:44:17.096673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:17.095992 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:17.265880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:17.255726 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" podStartSLOduration=3.255702851 podStartE2EDuration="3.255702851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:17.146479737 +0000 UTC m=+1222.574357669" watchObservedRunningTime="2026-11-01 22:44:17.255702851 +0000 UTC m=+1222.683580790" Nov 01 22:44:17.841759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:17.841725 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:18.348718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.348671 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.380754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.380707 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.423721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.421662 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.459601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.455659 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.638797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.638688 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:44:18.660413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.659987 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.661206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.660028 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:44:18.702812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.702722 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.736848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.736804 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.760837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.760795 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:44:18.761374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.761341 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:44:18.770961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.770927 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:18.781201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.781170 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:44:18.781673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:18.781650 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:44:19.134124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.134092 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:19.237646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.224671 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:44:19.251625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.248430 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:19.352609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.351683 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.352609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.352306 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:44:19.352609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.336029 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.402778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.402036 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.402778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.402623 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:44:19.479724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.479670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.505279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.505246 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:44:19.549799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.513468 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:44:19.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.660856 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.729444 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:44:19.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.729502 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:44:19.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.660888 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.682334 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.750679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.712129 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.781885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.757085 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:44:19.795759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.795726 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:44:19.893667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.782515 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:44:19.894099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.782545 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:44:19.931605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.788919 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.944639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.788984 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:44:19.944639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.789006 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:19.944639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.782172 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:44:19.963617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.962063 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:44:19.963617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.962138 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:44:19.963617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.962178 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:44:19.963617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.962214 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:44:19.963617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.962248 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:44:19.963617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.962283 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:44:19.963617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:19.962318 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:44:20.283600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.280138 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:20.283600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.280807 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:44:20.340602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.339858 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:20.376116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.376071 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.418728 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.559645 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.559681 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.799560 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.835623 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.010283 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.010325 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.010358 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.010391 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.010423 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.943762 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.986117 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:44:21.017638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:20.986155 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:44:21.028562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.028526 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:44:21.162648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.162618 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:44:21.163632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.156445 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:21.180052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.180021 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp"] Nov 01 22:44:21.222824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.222780 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f854ec-td7kp"] Nov 01 22:44:21.242036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.229299 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:44:21.302863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.302778 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdfd5b3-f010-40ef-91c0-7cb5a16e801a" path="/var/lib/kubelet/pods/9cdfd5b3-f010-40ef-91c0-7cb5a16e801a/volumes" Nov 01 22:44:21.303756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.303731 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:44:21.304010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.303993 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:44:21.304204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.304189 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:44:21.304433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.304361 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:44:21.304852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.304835 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:44:21.318081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.310857 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:21.325666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.324562 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:44:21.329279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:21.329002 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:44:22.189781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:22.189718 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:22.919420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:22.919388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:23.722361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:23.722329 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 22:44:23.762271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:23.762235 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:24.517937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:24.517906 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:25.251623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:25.250974 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:25.871889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:25.871861 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:26.034213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:26.034185 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2"] Nov 01 22:44:26.042540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:26.042511 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb55705d-bpcv2"] Nov 01 22:44:26.694904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:26.694866 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:26.737409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:26.737380 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35583d35-83a2-4e3d-9997-6773a898f70f" path="/var/lib/kubelet/pods/35583d35-83a2-4e3d-9997-6773a898f70f/volumes" Nov 01 22:44:26.797857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:26.797829 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:27.234825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:27.234792 4981 generic.go:356] "Generic (PLEG): container finished" podID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerID="d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768" exitCode=0 Nov 01 22:44:27.235004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:27.234874 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" event={"ID":"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8","Type":"ContainerDied","Data":"d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768"} Nov 01 22:44:27.235377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:27.235361 4981 scope.go:122] "RemoveContainer" containerID="d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768" Nov 01 22:44:27.237172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:27.237146 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:44:27.237252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:27.237215 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:44:27.237252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:27.237248 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:44:28.036253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:28.036215 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:28.248960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:28.248924 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" event={"ID":"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8","Type":"ContainerStarted","Data":"6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d"} Nov 01 22:44:28.969779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:28.969744 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:29.020101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.019838 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:44:29.022700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.022659 4981 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 01 22:44:29.022888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.022770 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 01 22:44:29.058081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.058047 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr"] Nov 01 22:44:29.064932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.064902 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc63a4d1-2c2nr"] Nov 01 22:44:29.405558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.405467 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:44:29.803110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.803075 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:29.901402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:29.901373 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:30.008619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:30.008566 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:30.019252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:30.018953 4981 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 01 22:44:30.019252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:30.019023 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 01 22:44:30.737616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:30.737557 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad69459-832e-4a63-b7a3-3fb5cd482c4b" path="/var/lib/kubelet/pods/aad69459-832e-4a63-b7a3-3fb5cd482c4b/volumes" Nov 01 22:44:30.845240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:30.845206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:31.021120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:31.020538 4981 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 01 22:44:31.021120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:31.020620 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 01 22:44:31.755504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:31.755466 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:32.017516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:32.017434 4981 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 01 22:44:32.017516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:32.017498 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 01 22:44:32.462018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:32.461987 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:33.017167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:33.017129 4981 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 01 22:44:33.017616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:33.017196 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 01 22:44:33.041228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:33.041191 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws"] Nov 01 22:44:33.047327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:33.047299 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74a5e869-bq7ws"] Nov 01 22:44:33.157590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:33.157539 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:33.871981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:33.871946 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:34.025608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.019563 4981 patch_prober.go:29] interesting pod/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 01 22:44:34.025608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.019657 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 01 22:44:34.037309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.037280 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5"] Nov 01 22:44:34.049473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.049442 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2346a4fe-cggn5"] Nov 01 22:44:34.164722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.164640 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:44:34.661985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.661951 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:34.753091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.753059 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9266fac5-94fb-4ba1-82f2-fe7131f1b0de" path="/var/lib/kubelet/pods/9266fac5-94fb-4ba1-82f2-fe7131f1b0de/volumes" Nov 01 22:44:34.753846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:34.753806 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6" path="/var/lib/kubelet/pods/93d8b2dd-b1b9-4b3a-abc4-7538e938bbf6/volumes" Nov 01 22:44:35.023530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:35.023495 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:44:35.024852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:35.024828 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:44:35.618370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:35.618337 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:36.027316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:36.027286 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx"] Nov 01 22:44:36.033855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:36.033830 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca42470-rw4kx"] Nov 01 22:44:36.392571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:36.392495 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:36.740721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:36.740686 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b2dfc48-2818-4dc9-a3e2-0bac45112c37" path="/var/lib/kubelet/pods/5b2dfc48-2818-4dc9-a3e2-0bac45112c37/volumes" Nov 01 22:44:37.028452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:37.028364 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7"] Nov 01 22:44:37.034266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:37.034237 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c8e53b-f2pz7"] Nov 01 22:44:37.097620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:37.097571 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:37.868891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:37.868861 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:38.036651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:38.036617 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n"] Nov 01 22:44:38.037724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:38.037702 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b34ab10-6gx7n"] Nov 01 22:44:38.267890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:38.267825 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:38.683498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:38.683472 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:38.768321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:38.768269 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02e814b-1913-428b-b40e-f8c5ecee320b" path="/var/lib/kubelet/pods/b02e814b-1913-428b-b40e-f8c5ecee320b/volumes" Nov 01 22:44:38.768827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:38.768809 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33037fa-36a4-4475-bc97-a573ac90893d" path="/var/lib/kubelet/pods/c33037fa-36a4-4475-bc97-a573ac90893d/volumes" Nov 01 22:44:39.459723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:39.459693 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:40.193744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:40.193710 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:40.871410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:40.871377 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:41.137374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.137300 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7"] Nov 01 22:44:41.144044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.144023 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.151274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.151248 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:44:41.152706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.152682 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:41.152706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.152685 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:44:41.152913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.152898 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:44:41.154069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.154046 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7"] Nov 01 22:44:41.307713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.307672 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.307713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.307713 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pzw\" (UniqueName: \"kubernetes.io/projected/8459768c-6d04-42c7-a464-a6d73092db73-kube-api-access-46pzw\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.307969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.307946 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.409815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.409717 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.410043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.410026 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.410090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.410057 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pzw\" (UniqueName: \"kubernetes.io/projected/8459768c-6d04-42c7-a464-a6d73092db73-kube-api-access-46pzw\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.410425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.410405 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.412344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.412318 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.419519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.419496 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pzw\" (UniqueName: \"kubernetes.io/projected/8459768c-6d04-42c7-a464-a6d73092db73-kube-api-access-46pzw\") pod \"maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.463322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.463292 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:41.897887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:41.897857 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7"] Nov 01 22:44:42.082763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:42.082735 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:42.423152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:42.423114 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" event={"ID":"8459768c-6d04-42c7-a464-a6d73092db73","Type":"ContainerStarted","Data":"8911d152f0481c48cb6c6e80a6ad5aff5d9f20cd74f9cc68d9c9d6d9808909ad"} Nov 01 22:44:42.423329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:42.423180 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" event={"ID":"8459768c-6d04-42c7-a464-a6d73092db73","Type":"ContainerStarted","Data":"cb795bf4784e5dad8b2bb94a61e7864b76c2c726aab214a4242f3f7fe943005c"} Nov 01 22:44:42.445755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:42.445705 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" podStartSLOduration=1.445690677 podStartE2EDuration="1.445690677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:42.440541634 +0000 UTC m=+1247.868419566" watchObservedRunningTime="2026-11-01 22:44:42.445690677 +0000 UTC m=+1247.873568607" Nov 01 22:44:42.803319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:42.803289 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:43.434719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:43.434681 4981 generic.go:356] "Generic (PLEG): container finished" podID="8459768c-6d04-42c7-a464-a6d73092db73" containerID="8911d152f0481c48cb6c6e80a6ad5aff5d9f20cd74f9cc68d9c9d6d9808909ad" exitCode=0 Nov 01 22:44:43.435135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:43.434773 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" event={"ID":"8459768c-6d04-42c7-a464-a6d73092db73","Type":"ContainerDied","Data":"8911d152f0481c48cb6c6e80a6ad5aff5d9f20cd74f9cc68d9c9d6d9808909ad"} Nov 01 22:44:43.677676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:43.677637 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:44.718664 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:44:44.717842 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:44:44.892439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.892408 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:44.935655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.935228 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-xks-ca\") pod \"8459768c-6d04-42c7-a464-a6d73092db73\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " Nov 01 22:44:44.935655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.935306 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-openshift-ca\") pod \"8459768c-6d04-42c7-a464-a6d73092db73\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " Nov 01 22:44:44.935655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.935387 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8459768c-6d04-42c7-a464-a6d73092db73-kube-api-access-46pzw\" (UniqueName: \"kubernetes.io/projected/8459768c-6d04-42c7-a464-a6d73092db73-kube-api-access-46pzw\") pod \"8459768c-6d04-42c7-a464-a6d73092db73\" (UID: \"8459768c-6d04-42c7-a464-a6d73092db73\") " Nov 01 22:44:44.935966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.935766 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-openshift-ca" pod "8459768c-6d04-42c7-a464-a6d73092db73" (UID: "8459768c-6d04-42c7-a464-a6d73092db73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:44.936345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.936313 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:44.938069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.938040 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-xks-ca" pod "8459768c-6d04-42c7-a464-a6d73092db73" (UID: "8459768c-6d04-42c7-a464-a6d73092db73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:44:44.938661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.938638 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8459768c-6d04-42c7-a464-a6d73092db73-kube-api-access-46pzw" pod "8459768c-6d04-42c7-a464-a6d73092db73" (UID: "8459768c-6d04-42c7-a464-a6d73092db73"). InnerVolumeSpecName "kube-api-access-46pzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:44.973667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:44.970227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:45.038188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.038093 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459768c-6d04-42c7-a464-a6d73092db73-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.038188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.038130 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pzw\" (UniqueName: \"kubernetes.io/projected/8459768c-6d04-42c7-a464-a6d73092db73-kube-api-access-46pzw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:45.243643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.243509 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:45.460507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.460474 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" Nov 01 22:44:45.460507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.460490 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7" event={"ID":"8459768c-6d04-42c7-a464-a6d73092db73","Type":"ContainerDied","Data":"cb795bf4784e5dad8b2bb94a61e7864b76c2c726aab214a4242f3f7fe943005c"} Nov 01 22:44:45.460783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.460550 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb795bf4784e5dad8b2bb94a61e7864b76c2c726aab214a4242f3f7fe943005c" Nov 01 22:44:45.577799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.577725 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:45.890448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:45.890369 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:46.054950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:46.054909 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:46.194641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:46.192923 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:46.887231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:46.887192 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:47.480514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:47.480479 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:47.689513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:47.689479 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:48.299964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:48.299890 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:48.423561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:48.423525 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:48.558408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:48.558332 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:48.868005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:48.867930 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:49.632453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:49.632423 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:50.426885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:50.426851 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:50.588396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:50.588361 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:50.771864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:50.771772 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:51.487763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:51.487728 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92"] Nov 01 22:44:51.595445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:51.595411 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:51.756179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:51.756100 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:52.270943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:52.270909 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:52.423710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:52.423663 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:52.651765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:52.649023 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:53.776598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:53.776547 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:53.883439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:53.883374 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:54.367091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.367057 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:54.557519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.557440 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q"] Nov 01 22:44:54.560163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.560135 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8459768c-6d04-42c7-a464-a6d73092db73" containerName="revoke-keys" Nov 01 22:44:54.560348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.560332 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8459768c-6d04-42c7-a464-a6d73092db73" containerName="revoke-keys" Nov 01 22:44:54.560674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.560654 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8459768c-6d04-42c7-a464-a6d73092db73" containerName="revoke-keys" Nov 01 22:44:54.565564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.565546 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:54.579857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.579831 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:54.581335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.581134 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:44:54.581335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.581198 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:44:54.581335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.581300 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:44:54.612950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.612916 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q"] Nov 01 22:44:54.661659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.643518 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:54.661659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.643737 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:54.661659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.643851 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxf5p\" (UniqueName: \"kubernetes.io/projected/2754c0df-eb00-4688-b7eb-112aa189c0fa-kube-api-access-wxf5p\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:54.998485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:54.979550 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:55.000741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.000700 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxf5p\" (UniqueName: \"kubernetes.io/projected/2754c0df-eb00-4688-b7eb-112aa189c0fa-kube-api-access-wxf5p\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:55.001279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.001243 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:55.169601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.162173 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:44:55.169601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.162280 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:44:55.187885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.187851 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:44:55.223602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.211550 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:55.509598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.475205 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:55.509598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.508283 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxf5p\" (UniqueName: \"kubernetes.io/projected/2754c0df-eb00-4688-b7eb-112aa189c0fa-kube-api-access-wxf5p\") pod \"maas-api-revoke-sub-980bfa4e-txk8q\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:55.578600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.578128 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:44:55.655603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.638073 4981 scope.go:122] "RemoveContainer" containerID="e4a54a3aa5587ac5f45f123fd0942f176d06595639b96dcc3588789f579cb5b1" Nov 01 22:44:55.709657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.705309 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:44:55.780537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.766781 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:55.982400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.955100 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 01 22:44:55.982400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.955413 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.033657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.998683 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:56.033657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:55.998772 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.107064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.107022 4981 scope.go:122] "RemoveContainer" containerID="0b163d04f3b42d778b6e20352e9fd72839a740546cefc2423c61c51c7362a8e6" Nov 01 22:44:56.107495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.107467 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:44:56.107732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.107689 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.117026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.108267 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.131629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.108304 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.131629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.108326 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:44:56.131629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.111131 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:44:56.137641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.131970 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.179642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.179347 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.214970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.214643 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.253815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.253757 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.322811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.322712 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.337691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.337651 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:44:56.338220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.338189 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.339383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.339355 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.426013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.425987 4981 scope.go:122] "RemoveContainer" containerID="9b1d9e9f4974b483ecaa9253bccc9ad5dd08b49357ecff3c3d862d8e02596721" Nov 01 22:44:56.450911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.450662 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:56.493435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.492648 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" podUID="1609f345-e130-47a0-a92d-e7ae55616848" containerName="payload-processing" containerID="cri-o://7ea15d31f267f39706d88cce66e2fa18fd4d6015ce551ba29180c30407f37fe5" gracePeriod=25 Nov 01 22:44:56.919599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.912495 4981 scope.go:122] "RemoveContainer" containerID="569ff80db8f626ff0aaabc2ac01cc2eca92a0979add6ad76e18856f18b1c9d43" Nov 01 22:44:56.919599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:56.919385 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.016909 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.015679 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.080231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.080190 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.084157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.084126 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.125630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.118667 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.136104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.132827 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.201779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.130026 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:44:57.201779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.200673 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.206425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.206199 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.373896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.224374 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.385786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.244551 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:44:57.402712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.257277 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.540340 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:44:57.303911 4981 logging.go:55] [core] [Channel #2084 SubChannel #2085]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.162:50052", ServerName: "10.134.0.162:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.162:50052: connect: connection refused" Nov 01 22:44:57.540340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.353209 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:44:57.540340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.428425 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.540340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.428696 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:44:57.644619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.644327 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:44:57.644975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.644959 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:44:57.645122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.645112 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:44:57.645249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.645239 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:44:57.645373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.645364 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:44:57.645494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.645485 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:44:57.645632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.645621 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:44:57.645769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.645758 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:44:57.645918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.645905 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:44:57.646075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.646064 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:44:57.646202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.646192 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:44:57.646329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.646320 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:44:57.646445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.646437 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:44:57.646562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.646553 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:44:57.646719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.646695 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:44:57.666335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.666168 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:44:57.680870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.679432 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:44:57.700155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.699024 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:57.700155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.699630 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:44:57.730745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.718926 4981 scope.go:122] "RemoveContainer" containerID="1d93497e3db6068ba20b6440bdaf5d4e477522f9b6600f614ef46d0b884ae246" Nov 01 22:44:57.769495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.734448 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:44:57.769495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.734507 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:44:57.769495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.744395 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q"] Nov 01 22:44:57.769495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.754680 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:44:57.769495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.754736 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:44:57.799264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.799233 4981 generic.go:356] "Generic (PLEG): container finished" podID="1609f345-e130-47a0-a92d-e7ae55616848" containerID="7ea15d31f267f39706d88cce66e2fa18fd4d6015ce551ba29180c30407f37fe5" exitCode=0 Nov 01 22:44:57.799414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.799385 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" event={"ID":"1609f345-e130-47a0-a92d-e7ae55616848","Type":"ContainerDied","Data":"7ea15d31f267f39706d88cce66e2fa18fd4d6015ce551ba29180c30407f37fe5"} Nov 01 22:44:57.819733 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:44:57.819702 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2754c0df_eb00_4688_b7eb_112aa189c0fa.slice/crio-93a7704e144a540d971766ea567dc59d8be04adb5c567e59db3f6e108f9fc183 WatchSource:0}: Error finding container 93a7704e144a540d971766ea567dc59d8be04adb5c567e59db3f6e108f9fc183: Status 404 returned error can't find the container with id 93a7704e144a540d971766ea567dc59d8be04adb5c567e59db3f6e108f9fc183 Nov 01 22:44:57.838917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:57.838894 4981 scope.go:122] "RemoveContainer" containerID="0672b9fd11f97ba2d29d699d96e6e37775919e5e5c1ebc5e18df80597987ff45" Nov 01 22:44:58.165941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.165917 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:44:58.245182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.245130 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" podUID="1609f345-e130-47a0-a92d-e7ae55616848" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.162:50052\" within 1s: context deadline exceeded" Nov 01 22:44:58.258138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.258104 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1609f345-e130-47a0-a92d-e7ae55616848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1609f345-e130-47a0-a92d-e7ae55616848-plugins-config-volume\") pod \"1609f345-e130-47a0-a92d-e7ae55616848\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " Nov 01 22:44:58.258361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.258343 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1609f345-e130-47a0-a92d-e7ae55616848-kube-api-access-b72fb\" (UniqueName: \"kubernetes.io/projected/1609f345-e130-47a0-a92d-e7ae55616848-kube-api-access-b72fb\") pod \"1609f345-e130-47a0-a92d-e7ae55616848\" (UID: \"1609f345-e130-47a0-a92d-e7ae55616848\") " Nov 01 22:44:58.258686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.258640 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1609f345-e130-47a0-a92d-e7ae55616848-plugins-config-volume" pod "1609f345-e130-47a0-a92d-e7ae55616848" (UID: "1609f345-e130-47a0-a92d-e7ae55616848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:44:58.259157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.259139 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1609f345-e130-47a0-a92d-e7ae55616848-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.262695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.262669 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1609f345-e130-47a0-a92d-e7ae55616848-kube-api-access-b72fb" pod "1609f345-e130-47a0-a92d-e7ae55616848" (UID: "1609f345-e130-47a0-a92d-e7ae55616848"). InnerVolumeSpecName "kube-api-access-b72fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:44:58.360400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.360366 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b72fb\" (UniqueName: \"kubernetes.io/projected/1609f345-e130-47a0-a92d-e7ae55616848-kube-api-access-b72fb\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:44:58.473509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.473472 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:58.813402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.813360 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" event={"ID":"1609f345-e130-47a0-a92d-e7ae55616848","Type":"ContainerDied","Data":"be1ec5f7e5487ac2f6ef743768adacb96cceb727cd7b270e01062dba5cf514a2"} Nov 01 22:44:58.813548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.813452 4981 scope.go:122] "RemoveContainer" containerID="7ea15d31f267f39706d88cce66e2fa18fd4d6015ce551ba29180c30407f37fe5" Nov 01 22:44:58.813548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.813375 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92" Nov 01 22:44:58.815624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.815549 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" event={"ID":"2754c0df-eb00-4688-b7eb-112aa189c0fa","Type":"ContainerStarted","Data":"b0e48db0cf269d874bb94aca2cfc2f64e95a4bd33271e3986ba379e634c44575"} Nov 01 22:44:58.815748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.815712 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" event={"ID":"2754c0df-eb00-4688-b7eb-112aa189c0fa","Type":"ContainerStarted","Data":"93a7704e144a540d971766ea567dc59d8be04adb5c567e59db3f6e108f9fc183"} Nov 01 22:44:58.908709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.908642 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:44:58.909360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.909335 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92"] Nov 01 22:44:58.917080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.917053 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4534ad-78dbf9c44f-b6h92"] Nov 01 22:44:58.957779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:58.957731 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" podStartSLOduration=4.957716625 podStartE2EDuration="4.957716625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:58.953060114 +0000 UTC m=+1264.380938071" watchObservedRunningTime="2026-11-01 22:44:58.957716625 +0000 UTC m=+1264.385594558" Nov 01 22:44:59.827295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:59.827261 4981 generic.go:356] "Generic (PLEG): container finished" podID="2754c0df-eb00-4688-b7eb-112aa189c0fa" containerID="b0e48db0cf269d874bb94aca2cfc2f64e95a4bd33271e3986ba379e634c44575" exitCode=0 Nov 01 22:44:59.827817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:44:59.827345 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" event={"ID":"2754c0df-eb00-4688-b7eb-112aa189c0fa","Type":"ContainerDied","Data":"b0e48db0cf269d874bb94aca2cfc2f64e95a4bd33271e3986ba379e634c44575"} Nov 01 22:45:00.032994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.032962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:00.037811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.037776 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk"] Nov 01 22:45:00.073481 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.073446 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d76e06a-v5skk"] Nov 01 22:45:00.192456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.192419 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc"] Nov 01 22:45:00.193846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.193817 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1609f345-e130-47a0-a92d-e7ae55616848" containerName="payload-processing" Nov 01 22:45:00.193846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.193844 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1609f345-e130-47a0-a92d-e7ae55616848" containerName="payload-processing" Nov 01 22:45:00.194075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.194062 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1609f345-e130-47a0-a92d-e7ae55616848" containerName="payload-processing" Nov 01 22:45:00.199457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.199426 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.205777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.205753 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp"] Nov 01 22:45:00.210845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.210820 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.212277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.212255 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq"] Nov 01 22:45:00.218232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.218201 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.228891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.224641 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp"] Nov 01 22:45:00.261274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.261237 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.295933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.295760 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4"] Nov 01 22:45:00.306504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.305000 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.314399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.314373 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc"] Nov 01 22:45:00.328613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.328537 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp"] Nov 01 22:45:00.329793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.329769 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/ce1efe76-4df3-492e-be84-4c58ad778677-kube-api-access-ndxbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.330508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.330470 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6wz\" (UniqueName: \"kubernetes.io/projected/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-kube-api-access-4n6wz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.330803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.330786 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1efe76-4df3-492e-be84-4c58ad778677-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.330977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.330964 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4st8n\" (UniqueName: \"kubernetes.io/projected/6fc6ede6-5a8f-400a-8557-287c62afcc01-kube-api-access-4st8n\") pod \"maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.331122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.331109 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6ede6-5a8f-400a-8557-287c62afcc01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.331292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.331273 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjtb\" (UniqueName: \"kubernetes.io/projected/dc92015e-f197-4c25-ba89-fafc04e006b8-kube-api-access-mkjtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.331497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.331480 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc92015e-f197-4c25-ba89-fafc04e006b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.332327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.332308 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.344859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.344787 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp"] Nov 01 22:45:00.357112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.357081 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq"] Nov 01 22:45:00.364018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.363982 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4"] Nov 01 22:45:00.371340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.371308 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn"] Nov 01 22:45:00.377433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.377411 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.381205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.381185 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg"] Nov 01 22:45:00.390750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.390727 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz"] Nov 01 22:45:00.391971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.390967 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.404153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.404130 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.417134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.416673 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4"] Nov 01 22:45:00.424485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.424460 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn"] Nov 01 22:45:00.424631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.424615 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.430346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.430330 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg"] Nov 01 22:45:00.434026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.433999 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6wz\" (UniqueName: \"kubernetes.io/projected/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-kube-api-access-4n6wz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.434228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.434207 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1efe76-4df3-492e-be84-4c58ad778677-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.434430 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.434411 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4st8n\" (UniqueName: \"kubernetes.io/projected/6fc6ede6-5a8f-400a-8557-287c62afcc01-kube-api-access-4st8n\") pod \"maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.434817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.434794 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs22\" (UniqueName: \"kubernetes.io/projected/eb47cc2a-e221-44bb-946c-7eca7267a452-kube-api-access-kbs22\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.435043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.435016 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6ede6-5a8f-400a-8557-287c62afcc01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.435769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.435745 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjtb\" (UniqueName: \"kubernetes.io/projected/dc92015e-f197-4c25-ba89-fafc04e006b8-kube-api-access-mkjtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.436015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.435997 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc92015e-f197-4c25-ba89-fafc04e006b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.436174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.436149 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24vv\" (UniqueName: \"kubernetes.io/projected/b1224a3d-870d-4c91-8508-ad97ad1cf806-kube-api-access-m24vv\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.436320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.436195 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.436320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.436269 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f25rp\" (UniqueName: \"kubernetes.io/projected/8472535c-7063-4bea-82b9-39a6ee919bc0-kube-api-access-f25rp\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.436799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.436371 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8472535c-7063-4bea-82b9-39a6ee919bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.436799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.436485 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47cc2a-e221-44bb-946c-7eca7267a452-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.436799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.436545 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1224a3d-870d-4c91-8508-ad97ad1cf806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.436799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.436660 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/ce1efe76-4df3-492e-be84-4c58ad778677-kube-api-access-ndxbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.437718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.437694 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1efe76-4df3-492e-be84-4c58ad778677-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.438015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.437993 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6ede6-5a8f-400a-8557-287c62afcc01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.438383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.438357 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.438473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.438460 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz"] Nov 01 22:45:00.438761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.438741 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc92015e-f197-4c25-ba89-fafc04e006b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.443472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.443418 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4"] Nov 01 22:45:00.451549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.451472 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4st8n\" (UniqueName: \"kubernetes.io/projected/6fc6ede6-5a8f-400a-8557-287c62afcc01-kube-api-access-4st8n\") pod \"maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.451719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.451697 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/ce1efe76-4df3-492e-be84-4c58ad778677-kube-api-access-ndxbp\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.454165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.454127 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjtb\" (UniqueName: \"kubernetes.io/projected/dc92015e-f197-4c25-ba89-fafc04e006b8-kube-api-access-mkjtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.455086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.455044 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6wz\" (UniqueName: \"kubernetes.io/projected/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-kube-api-access-4n6wz\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.457563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.457123 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd"] Nov 01 22:45:00.463803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.463785 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.469088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.469047 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx"] Nov 01 22:45:00.475549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.475520 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:00.475684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.475668 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd"] Nov 01 22:45:00.485853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.485826 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx"] Nov 01 22:45:00.538069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538033 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f25rp\" (UniqueName: \"kubernetes.io/projected/8472535c-7063-4bea-82b9-39a6ee919bc0-kube-api-access-f25rp\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.538203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538102 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rw2s\" (UniqueName: \"kubernetes.io/projected/05dfc6d0-c041-47aa-86e1-227e3b32843b-kube-api-access-2rw2s\") pod \"maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:00.538203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538136 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8472535c-7063-4bea-82b9-39a6ee919bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.538203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538158 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszft\" (UniqueName: \"kubernetes.io/projected/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-kube-api-access-zszft\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.538380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538342 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47cc2a-e221-44bb-946c-7eca7267a452-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.538444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538426 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1224a3d-870d-4c91-8508-ad97ad1cf806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.538546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538531 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.538695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538670 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40734f-e0ed-49ef-b552-8e29e6904fa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.538775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538743 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrg2v\" (UniqueName: \"kubernetes.io/projected/4f36e99e-f6ff-4007-a2ca-61792b5982bd-kube-api-access-hrg2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.538884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538860 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs22\" (UniqueName: \"kubernetes.io/projected/eb47cc2a-e221-44bb-946c-7eca7267a452-kube-api-access-kbs22\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.538928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538914 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8472535c-7063-4bea-82b9-39a6ee919bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.538987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538935 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36e99e-f6ff-4007-a2ca-61792b5982bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.539040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.538992 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqnqx\" (UniqueName: \"kubernetes.io/projected/6c40734f-e0ed-49ef-b552-8e29e6904fa3-kube-api-access-vqnqx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.539097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.539048 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47cc2a-e221-44bb-946c-7eca7267a452-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.539155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.539125 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24vv\" (UniqueName: \"kubernetes.io/projected/b1224a3d-870d-4c91-8508-ad97ad1cf806-kube-api-access-m24vv\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.539210 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.539168 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dfc6d0-c041-47aa-86e1-227e3b32843b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:00.539210 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.539194 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1224a3d-870d-4c91-8508-ad97ad1cf806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.546835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.546807 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f25rp\" (UniqueName: \"kubernetes.io/projected/8472535c-7063-4bea-82b9-39a6ee919bc0-kube-api-access-f25rp\") pod \"maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.547732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.547705 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs22\" (UniqueName: \"kubernetes.io/projected/eb47cc2a-e221-44bb-946c-7eca7267a452-kube-api-access-kbs22\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.548065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.548042 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24vv\" (UniqueName: \"kubernetes.io/projected/b1224a3d-870d-4c91-8508-ad97ad1cf806-kube-api-access-m24vv\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.608167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.608137 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:00.615506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.615480 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:00.619618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.619571 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:00.623237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.623219 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:00.632525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.632505 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:00.640467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.640439 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszft\" (UniqueName: \"kubernetes.io/projected/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-kube-api-access-zszft\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.640626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.640586 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.641405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.640687 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40734f-e0ed-49ef-b552-8e29e6904fa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.641405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.640718 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrg2v\" (UniqueName: \"kubernetes.io/projected/4f36e99e-f6ff-4007-a2ca-61792b5982bd-kube-api-access-hrg2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.641405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.640801 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36e99e-f6ff-4007-a2ca-61792b5982bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.641405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.640852 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqnqx\" (UniqueName: \"kubernetes.io/projected/6c40734f-e0ed-49ef-b552-8e29e6904fa3-kube-api-access-vqnqx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.641405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.640933 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dfc6d0-c041-47aa-86e1-227e3b32843b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:00.641405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.641036 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rw2s\" (UniqueName: \"kubernetes.io/projected/05dfc6d0-c041-47aa-86e1-227e3b32843b-kube-api-access-2rw2s\") pod \"maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:00.641717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.641463 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40734f-e0ed-49ef-b552-8e29e6904fa3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.641717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.641659 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36e99e-f6ff-4007-a2ca-61792b5982bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.642057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.642036 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dfc6d0-c041-47aa-86e1-227e3b32843b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:00.642612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.642562 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.654765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.652312 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqnqx\" (UniqueName: \"kubernetes.io/projected/6c40734f-e0ed-49ef-b552-8e29e6904fa3-kube-api-access-vqnqx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.654765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.652685 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszft\" (UniqueName: \"kubernetes.io/projected/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-kube-api-access-zszft\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.654765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.652979 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rw2s\" (UniqueName: \"kubernetes.io/projected/05dfc6d0-c041-47aa-86e1-227e3b32843b-kube-api-access-2rw2s\") pod \"maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:00.654765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.653939 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrg2v\" (UniqueName: \"kubernetes.io/projected/4f36e99e-f6ff-4007-a2ca-61792b5982bd-kube-api-access-hrg2v\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.726505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.725775 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:00.765738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.732757 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:00.765738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.752680 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:00.765738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.754740 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1609f345-e130-47a0-a92d-e7ae55616848" path="/var/lib/kubelet/pods/1609f345-e130-47a0-a92d-e7ae55616848/volumes" Nov 01 22:45:00.765738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.761971 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6a6046-9ea5-4564-ab01-02edf3ddcdb5" path="/var/lib/kubelet/pods/ee6a6046-9ea5-4564-ab01-02edf3ddcdb5/volumes" Nov 01 22:45:00.803103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.803068 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:00.815325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.815298 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:00.824864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:00.824690 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:01.128030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.127958 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:01.213844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.213732 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc"] Nov 01 22:45:01.234778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.234755 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp"] Nov 01 22:45:01.274001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.273975 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:45:01.325619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.325565 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-openshift-ca\") pod \"2754c0df-eb00-4688-b7eb-112aa189c0fa\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " Nov 01 22:45:01.325890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.325871 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-xks-ca\") pod \"2754c0df-eb00-4688-b7eb-112aa189c0fa\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " Nov 01 22:45:01.325974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.325889 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-openshift-ca" pod "2754c0df-eb00-4688-b7eb-112aa189c0fa" (UID: "2754c0df-eb00-4688-b7eb-112aa189c0fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:01.326029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.325985 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2754c0df-eb00-4688-b7eb-112aa189c0fa-kube-api-access-wxf5p\" (UniqueName: \"kubernetes.io/projected/2754c0df-eb00-4688-b7eb-112aa189c0fa-kube-api-access-wxf5p\") pod \"2754c0df-eb00-4688-b7eb-112aa189c0fa\" (UID: \"2754c0df-eb00-4688-b7eb-112aa189c0fa\") " Nov 01 22:45:01.326689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.326667 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:01.371890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.371847 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq"] Nov 01 22:45:01.379936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.379903 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp"] Nov 01 22:45:01.380445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.380417 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2754c0df-eb00-4688-b7eb-112aa189c0fa-kube-api-access-wxf5p" pod "2754c0df-eb00-4688-b7eb-112aa189c0fa" (UID: "2754c0df-eb00-4688-b7eb-112aa189c0fa"). InnerVolumeSpecName "kube-api-access-wxf5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:01.389055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.389019 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-xks-ca" pod "2754c0df-eb00-4688-b7eb-112aa189c0fa" (UID: "2754c0df-eb00-4688-b7eb-112aa189c0fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:01.394815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.394171 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc6ede6_5a8f_400a_8557_287c62afcc01.slice/crio-cfb4614f3e0d01c4fecfb7ed7ce5b6180f5a7623e1cd003dc5fafac779987553 WatchSource:0}: Error finding container cfb4614f3e0d01c4fecfb7ed7ce5b6180f5a7623e1cd003dc5fafac779987553: Status 404 returned error can't find the container with id cfb4614f3e0d01c4fecfb7ed7ce5b6180f5a7623e1cd003dc5fafac779987553 Nov 01 22:45:01.394815 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.394485 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc92015e_f197_4c25_ba89_fafc04e006b8.slice/crio-1ab08004fbf03181dbab44ecf5e0c192f66259d394a49861634068923122e983 WatchSource:0}: Error finding container 1ab08004fbf03181dbab44ecf5e0c192f66259d394a49861634068923122e983: Status 404 returned error can't find the container with id 1ab08004fbf03181dbab44ecf5e0c192f66259d394a49861634068923122e983 Nov 01 22:45:01.408618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.407487 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4"] Nov 01 22:45:01.425985 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.425928 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb47cc2a_e221_44bb_946c_7eca7267a452.slice/crio-9aab46f1d38eade117d57e561acfab1b99a1d2f290efbde9f27db417349e35b7 WatchSource:0}: Error finding container 9aab46f1d38eade117d57e561acfab1b99a1d2f290efbde9f27db417349e35b7: Status 404 returned error can't find the container with id 9aab46f1d38eade117d57e561acfab1b99a1d2f290efbde9f27db417349e35b7 Nov 01 22:45:01.428773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.428745 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2754c0df-eb00-4688-b7eb-112aa189c0fa-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:01.428923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.428905 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxf5p\" (UniqueName: \"kubernetes.io/projected/2754c0df-eb00-4688-b7eb-112aa189c0fa-kube-api-access-wxf5p\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:01.514205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.514175 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn"] Nov 01 22:45:01.541991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.541958 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz"] Nov 01 22:45:01.569128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.569101 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg"] Nov 01 22:45:01.678720 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.678676 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1224a3d_870d_4c91_8508_ad97ad1cf806.slice/crio-12ff3b4deae33cb8f1f5d493a2821ed7cc547eeb3cd531604ae982bb8544c95d WatchSource:0}: Error finding container 12ff3b4deae33cb8f1f5d493a2821ed7cc547eeb3cd531604ae982bb8544c95d: Status 404 returned error can't find the container with id 12ff3b4deae33cb8f1f5d493a2821ed7cc547eeb3cd531604ae982bb8544c95d Nov 01 22:45:01.681080 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.681037 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3a5cbc_b1bd_4ca3_9f69_af0bff0cefd3.slice/crio-5fe0a3ee231326a0fe93df025a665188c539bb27951f7687b7464945e8a54a7d WatchSource:0}: Error finding container 5fe0a3ee231326a0fe93df025a665188c539bb27951f7687b7464945e8a54a7d: Status 404 returned error can't find the container with id 5fe0a3ee231326a0fe93df025a665188c539bb27951f7687b7464945e8a54a7d Nov 01 22:45:01.707889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.707859 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4"] Nov 01 22:45:01.713877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.713853 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd"] Nov 01 22:45:01.719386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.719358 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx"] Nov 01 22:45:01.777103 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.777016 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8472535c_7063_4bea_82b9_39a6ee919bc0.slice/crio-efda556e8e22f6233ff6680b92e60384366af2c5d032787040d6209b2086d9bf WatchSource:0}: Error finding container efda556e8e22f6233ff6680b92e60384366af2c5d032787040d6209b2086d9bf: Status 404 returned error can't find the container with id efda556e8e22f6233ff6680b92e60384366af2c5d032787040d6209b2086d9bf Nov 01 22:45:01.780675 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.780647 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f36e99e_f6ff_4007_a2ca_61792b5982bd.slice/crio-c7f71d754da33a9d25939cb42e019b670cce92e8c5cc129f4775a6997a18908e WatchSource:0}: Error finding container c7f71d754da33a9d25939cb42e019b670cce92e8c5cc129f4775a6997a18908e: Status 404 returned error can't find the container with id c7f71d754da33a9d25939cb42e019b670cce92e8c5cc129f4775a6997a18908e Nov 01 22:45:01.782338 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.782304 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c40734f_e0ed_49ef_b552_8e29e6904fa3.slice/crio-af2a24fc9cd57db2c8d4f181ce22cf0eb7d09fe7e77caffd9a54cb3e72d65fb8 WatchSource:0}: Error finding container af2a24fc9cd57db2c8d4f181ce22cf0eb7d09fe7e77caffd9a54cb3e72d65fb8: Status 404 returned error can't find the container with id af2a24fc9cd57db2c8d4f181ce22cf0eb7d09fe7e77caffd9a54cb3e72d65fb8 Nov 01 22:45:01.784154 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:01.784133 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dfc6d0_c041_47aa_86e1_227e3b32843b.slice/crio-edcc02d6116ffba6c5d17c137aef6e7d5d91e65324ebf752bca549a1331eb32c WatchSource:0}: Error finding container edcc02d6116ffba6c5d17c137aef6e7d5d91e65324ebf752bca549a1331eb32c: Status 404 returned error can't find the container with id edcc02d6116ffba6c5d17c137aef6e7d5d91e65324ebf752bca549a1331eb32c Nov 01 22:45:01.853602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.853558 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" event={"ID":"dc92015e-f197-4c25-ba89-fafc04e006b8","Type":"ContainerStarted","Data":"1ab08004fbf03181dbab44ecf5e0c192f66259d394a49861634068923122e983"} Nov 01 22:45:01.856062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.856026 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerStarted","Data":"c7f71d754da33a9d25939cb42e019b670cce92e8c5cc129f4775a6997a18908e"} Nov 01 22:45:01.858347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.858284 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" event={"ID":"cd3eb503-df56-4cad-ad04-91b08c7f0f6b","Type":"ContainerStarted","Data":"64acb50598114fce0d514cdd69b35be933c68203a0b5d120cbada58797d1f314"} Nov 01 22:45:01.860403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.860381 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" event={"ID":"2754c0df-eb00-4688-b7eb-112aa189c0fa","Type":"ContainerDied","Data":"93a7704e144a540d971766ea567dc59d8be04adb5c567e59db3f6e108f9fc183"} Nov 01 22:45:01.860544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.860528 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93a7704e144a540d971766ea567dc59d8be04adb5c567e59db3f6e108f9fc183" Nov 01 22:45:01.860666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.860393 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q" Nov 01 22:45:01.863150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.863119 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" event={"ID":"6c40734f-e0ed-49ef-b552-8e29e6904fa3","Type":"ContainerStarted","Data":"af2a24fc9cd57db2c8d4f181ce22cf0eb7d09fe7e77caffd9a54cb3e72d65fb8"} Nov 01 22:45:01.864901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.864878 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" event={"ID":"6fc6ede6-5a8f-400a-8557-287c62afcc01","Type":"ContainerStarted","Data":"cfb4614f3e0d01c4fecfb7ed7ce5b6180f5a7623e1cd003dc5fafac779987553"} Nov 01 22:45:01.866542 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.866515 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerStarted","Data":"12ff3b4deae33cb8f1f5d493a2821ed7cc547eeb3cd531604ae982bb8544c95d"} Nov 01 22:45:01.871065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.870897 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerStarted","Data":"9aab46f1d38eade117d57e561acfab1b99a1d2f290efbde9f27db417349e35b7"} Nov 01 22:45:01.875079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.875051 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerStarted","Data":"edcc02d6116ffba6c5d17c137aef6e7d5d91e65324ebf752bca549a1331eb32c"} Nov 01 22:45:01.876591 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.876552 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" event={"ID":"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3","Type":"ContainerStarted","Data":"5fe0a3ee231326a0fe93df025a665188c539bb27951f7687b7464945e8a54a7d"} Nov 01 22:45:01.877977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.877956 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" event={"ID":"ce1efe76-4df3-492e-be84-4c58ad778677","Type":"ContainerStarted","Data":"35b62196cce61f48bdca9fd866314e5f88dfa3790038c19c368925a2b3300d2a"} Nov 01 22:45:01.880069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:01.880041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerStarted","Data":"efda556e8e22f6233ff6680b92e60384366af2c5d032787040d6209b2086d9bf"} Nov 01 22:45:02.034804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.034720 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc"] Nov 01 22:45:02.038953 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.038921 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2545daf8-jpvsc"] Nov 01 22:45:02.097220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.097185 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:02.567856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.567763 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:45:02.567856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.567832 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:02.742086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.742042 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e66f661-71b2-4950-a9fc-ed1c9532454a" path="/var/lib/kubelet/pods/7e66f661-71b2-4950-a9fc-ed1c9532454a/volumes" Nov 01 22:45:02.896238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.896197 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" event={"ID":"6c40734f-e0ed-49ef-b552-8e29e6904fa3","Type":"ContainerStarted","Data":"9c320d725a9bcf76d590d89adfebb306e3a7e959b39d2fda8fc39b2b055ca041"} Nov 01 22:45:02.898226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.898198 4981 generic.go:356] "Generic (PLEG): container finished" podID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerID="7be8aeba0d3143d4f6e7c86c5c7fb683f6d1050adf7e6936163fb7a0765b432f" exitCode=60 Nov 01 22:45:02.898361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.898298 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" event={"ID":"6fc6ede6-5a8f-400a-8557-287c62afcc01","Type":"ContainerDied","Data":"7be8aeba0d3143d4f6e7c86c5c7fb683f6d1050adf7e6936163fb7a0765b432f"} Nov 01 22:45:02.898644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.898624 4981 scope.go:122] "RemoveContainer" containerID="7be8aeba0d3143d4f6e7c86c5c7fb683f6d1050adf7e6936163fb7a0765b432f" Nov 01 22:45:02.900486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.900459 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerStarted","Data":"0cd273544de30ab0ca3d21e7db2af5500200c12a1f9d447eef245afa5df6adce"} Nov 01 22:45:02.903415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.902904 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerStarted","Data":"8f4e5b13adb8e05837e3a96ebdc8198db3a8acf9a9a13640b668e06a80c9cd53"} Nov 01 22:45:02.904757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.904730 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerStarted","Data":"63457c08cc556c598870b145a8b924118d6882d20debdb25f27cbe7b82dd9ce1"} Nov 01 22:45:02.906556 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.906530 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" event={"ID":"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3","Type":"ContainerStarted","Data":"08770bd1f5da5fa43a8527d3953fd24900c3e21f9bf58a8da290f4bd61d4a762"} Nov 01 22:45:02.909243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.909217 4981 generic.go:356] "Generic (PLEG): container finished" podID="ce1efe76-4df3-492e-be84-4c58ad778677" containerID="7297a7753121b54c4cc7897cd82867701bf68a1a33f9b23db12a3472ae2457a5" exitCode=60 Nov 01 22:45:02.909369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.909343 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" event={"ID":"ce1efe76-4df3-492e-be84-4c58ad778677","Type":"ContainerDied","Data":"7297a7753121b54c4cc7897cd82867701bf68a1a33f9b23db12a3472ae2457a5"} Nov 01 22:45:02.909551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.909526 4981 scope.go:122] "RemoveContainer" containerID="7297a7753121b54c4cc7897cd82867701bf68a1a33f9b23db12a3472ae2457a5" Nov 01 22:45:02.911399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.911376 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerStarted","Data":"834fa2af44a3cb1036162b66a0f6330cfd4ec6a8d0537c7837e2409fdf28905c"} Nov 01 22:45:02.913173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.913153 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" event={"ID":"dc92015e-f197-4c25-ba89-fafc04e006b8","Type":"ContainerStarted","Data":"1543fc6db5aad1dce402f347cd03b986a5b12f904f43f280fed1e9dba17a8322"} Nov 01 22:45:02.915636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.915050 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerStarted","Data":"53baeaefc4348c97da46a2ba7f37cea739d5b7a1685fcda90c79e70e2de34dc2"} Nov 01 22:45:02.917473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.917413 4981 generic.go:356] "Generic (PLEG): container finished" podID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerID="ea648bf767c633557a377e3b46e0271e57afad461380bed530141c3c35377c0e" exitCode=60 Nov 01 22:45:02.917615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.917546 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" event={"ID":"cd3eb503-df56-4cad-ad04-91b08c7f0f6b","Type":"ContainerDied","Data":"ea648bf767c633557a377e3b46e0271e57afad461380bed530141c3c35377c0e"} Nov 01 22:45:02.917868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.917848 4981 scope.go:122] "RemoveContainer" containerID="ea648bf767c633557a377e3b46e0271e57afad461380bed530141c3c35377c0e" Nov 01 22:45:02.918862 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.918814 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" podStartSLOduration=2.91834756 podStartE2EDuration="2.91834756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.917921351 +0000 UTC m=+1268.345799283" watchObservedRunningTime="2026-11-01 22:45:02.91834756 +0000 UTC m=+1268.346225492" Nov 01 22:45:02.972779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:02.972723 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" podStartSLOduration=2.972703572 podStartE2EDuration="2.972703572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.964876066 +0000 UTC m=+1268.392753998" watchObservedRunningTime="2026-11-01 22:45:02.972703572 +0000 UTC m=+1268.400581505" Nov 01 22:45:03.000300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.000254 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" podStartSLOduration=3.000240239 podStartE2EDuration="3.000240239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.995046897 +0000 UTC m=+1268.422924828" watchObservedRunningTime="2026-11-01 22:45:03.000240239 +0000 UTC m=+1268.428118241" Nov 01 22:45:03.026809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.021473 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" podStartSLOduration=3.021452987 podStartE2EDuration="3.021452987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.014989969 +0000 UTC m=+1268.442867902" watchObservedRunningTime="2026-11-01 22:45:03.021452987 +0000 UTC m=+1268.449330920" Nov 01 22:45:03.039107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.039030 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" podStartSLOduration=3.03901131 podStartE2EDuration="3.03901131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.037785757 +0000 UTC m=+1268.465663688" watchObservedRunningTime="2026-11-01 22:45:03.03901131 +0000 UTC m=+1268.466889243" Nov 01 22:45:03.061478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.061421 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" podStartSLOduration=3.06140564 podStartE2EDuration="3.06140564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.05881477 +0000 UTC m=+1268.486692714" watchObservedRunningTime="2026-11-01 22:45:03.06140564 +0000 UTC m=+1268.489283572" Nov 01 22:45:03.100444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.100344 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" podStartSLOduration=3.100328046 podStartE2EDuration="3.100328046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.092231797 +0000 UTC m=+1268.520109729" watchObservedRunningTime="2026-11-01 22:45:03.100328046 +0000 UTC m=+1268.528205976" Nov 01 22:45:03.140902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.140847 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" podStartSLOduration=3.140831994 podStartE2EDuration="3.140831994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.136732741 +0000 UTC m=+1268.564610672" watchObservedRunningTime="2026-11-01 22:45:03.140831994 +0000 UTC m=+1268.568709924" Nov 01 22:45:03.329530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.329435 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:03.944548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.944453 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerDied","Data":"63457c08cc556c598870b145a8b924118d6882d20debdb25f27cbe7b82dd9ce1"} Nov 01 22:45:03.945051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.944609 4981 generic.go:356] "Generic (PLEG): container finished" podID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerID="63457c08cc556c598870b145a8b924118d6882d20debdb25f27cbe7b82dd9ce1" exitCode=60 Nov 01 22:45:03.945421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.945401 4981 scope.go:122] "RemoveContainer" containerID="63457c08cc556c598870b145a8b924118d6882d20debdb25f27cbe7b82dd9ce1" Nov 01 22:45:03.950163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.947770 4981 generic.go:356] "Generic (PLEG): container finished" podID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerID="08770bd1f5da5fa43a8527d3953fd24900c3e21f9bf58a8da290f4bd61d4a762" exitCode=60 Nov 01 22:45:03.950163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.947872 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" event={"ID":"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3","Type":"ContainerDied","Data":"08770bd1f5da5fa43a8527d3953fd24900c3e21f9bf58a8da290f4bd61d4a762"} Nov 01 22:45:03.950163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.948208 4981 scope.go:122] "RemoveContainer" containerID="08770bd1f5da5fa43a8527d3953fd24900c3e21f9bf58a8da290f4bd61d4a762" Nov 01 22:45:03.952149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.952117 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" event={"ID":"ce1efe76-4df3-492e-be84-4c58ad778677","Type":"ContainerStarted","Data":"2f6be5f9a4bbd83aaadb68a082787df50816404a40e827109a2e20570ba8e8a0"} Nov 01 22:45:03.954291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.954266 4981 generic.go:356] "Generic (PLEG): container finished" podID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerID="834fa2af44a3cb1036162b66a0f6330cfd4ec6a8d0537c7837e2409fdf28905c" exitCode=60 Nov 01 22:45:03.954419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.954365 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerDied","Data":"834fa2af44a3cb1036162b66a0f6330cfd4ec6a8d0537c7837e2409fdf28905c"} Nov 01 22:45:03.954651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.954605 4981 scope.go:122] "RemoveContainer" containerID="834fa2af44a3cb1036162b66a0f6330cfd4ec6a8d0537c7837e2409fdf28905c" Nov 01 22:45:03.956204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.956182 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerID="1543fc6db5aad1dce402f347cd03b986a5b12f904f43f280fed1e9dba17a8322" exitCode=60 Nov 01 22:45:03.956291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.956263 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" event={"ID":"dc92015e-f197-4c25-ba89-fafc04e006b8","Type":"ContainerDied","Data":"1543fc6db5aad1dce402f347cd03b986a5b12f904f43f280fed1e9dba17a8322"} Nov 01 22:45:03.956447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.956431 4981 scope.go:122] "RemoveContainer" containerID="1543fc6db5aad1dce402f347cd03b986a5b12f904f43f280fed1e9dba17a8322" Nov 01 22:45:03.957996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.957975 4981 generic.go:356] "Generic (PLEG): container finished" podID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerID="53baeaefc4348c97da46a2ba7f37cea739d5b7a1685fcda90c79e70e2de34dc2" exitCode=60 Nov 01 22:45:03.958071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.958041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerDied","Data":"53baeaefc4348c97da46a2ba7f37cea739d5b7a1685fcda90c79e70e2de34dc2"} Nov 01 22:45:03.958275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.958261 4981 scope.go:122] "RemoveContainer" containerID="53baeaefc4348c97da46a2ba7f37cea739d5b7a1685fcda90c79e70e2de34dc2" Nov 01 22:45:03.960375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.960353 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" event={"ID":"cd3eb503-df56-4cad-ad04-91b08c7f0f6b","Type":"ContainerStarted","Data":"ce99e929752c1e7d4315f266c122be5b9c7714f2983ea24efbd9ce9aa771b435"} Nov 01 22:45:03.962301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.962278 4981 generic.go:356] "Generic (PLEG): container finished" podID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerID="9c320d725a9bcf76d590d89adfebb306e3a7e959b39d2fda8fc39b2b055ca041" exitCode=60 Nov 01 22:45:03.962436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.962307 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" event={"ID":"6c40734f-e0ed-49ef-b552-8e29e6904fa3","Type":"ContainerDied","Data":"9c320d725a9bcf76d590d89adfebb306e3a7e959b39d2fda8fc39b2b055ca041"} Nov 01 22:45:03.962588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.962556 4981 scope.go:122] "RemoveContainer" containerID="9c320d725a9bcf76d590d89adfebb306e3a7e959b39d2fda8fc39b2b055ca041" Nov 01 22:45:03.965181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.965121 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" event={"ID":"6fc6ede6-5a8f-400a-8557-287c62afcc01","Type":"ContainerStarted","Data":"d10d696009be5de7b8a91614971c3ee477d7e4b056b267abc02701b88c54edcb"} Nov 01 22:45:03.970891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.970861 4981 generic.go:356] "Generic (PLEG): container finished" podID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerID="0cd273544de30ab0ca3d21e7db2af5500200c12a1f9d447eef245afa5df6adce" exitCode=60 Nov 01 22:45:03.971320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.971297 4981 scope.go:122] "RemoveContainer" containerID="0cd273544de30ab0ca3d21e7db2af5500200c12a1f9d447eef245afa5df6adce" Nov 01 22:45:03.971796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.971770 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerDied","Data":"0cd273544de30ab0ca3d21e7db2af5500200c12a1f9d447eef245afa5df6adce"} Nov 01 22:45:03.976267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.976243 4981 generic.go:356] "Generic (PLEG): container finished" podID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerID="8f4e5b13adb8e05837e3a96ebdc8198db3a8acf9a9a13640b668e06a80c9cd53" exitCode=60 Nov 01 22:45:03.976373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.976292 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerDied","Data":"8f4e5b13adb8e05837e3a96ebdc8198db3a8acf9a9a13640b668e06a80c9cd53"} Nov 01 22:45:03.976674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.976654 4981 scope.go:122] "RemoveContainer" containerID="8f4e5b13adb8e05837e3a96ebdc8198db3a8acf9a9a13640b668e06a80c9cd53" Nov 01 22:45:03.993272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:03.993223 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" podStartSLOduration=3.993209747 podStartE2EDuration="3.993209747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:03.991418202 +0000 UTC m=+1269.419296134" watchObservedRunningTime="2026-11-01 22:45:03.993209747 +0000 UTC m=+1269.421087677" Nov 01 22:45:04.170755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:04.170688 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" podStartSLOduration=4.170667793 podStartE2EDuration="4.170667793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.127704317 +0000 UTC m=+1269.555582249" watchObservedRunningTime="2026-11-01 22:45:04.170667793 +0000 UTC m=+1269.598545724" Nov 01 22:45:04.191265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:04.191215 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" podStartSLOduration=4.191200394 podStartE2EDuration="4.191200394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:04.185610556 +0000 UTC m=+1269.613488490" watchObservedRunningTime="2026-11-01 22:45:04.191200394 +0000 UTC m=+1269.619078377" Nov 01 22:45:04.300471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:04.300388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:04.823254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:04.823214 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:05.000752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.000717 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" event={"ID":"6c40734f-e0ed-49ef-b552-8e29e6904fa3","Type":"ContainerStarted","Data":"46f9afbce6c0aebd26bb5971e4b3166c6d55090e65b93ab6172e51f8a803f276"} Nov 01 22:45:05.003220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.003193 4981 generic.go:356] "Generic (PLEG): container finished" podID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerID="d10d696009be5de7b8a91614971c3ee477d7e4b056b267abc02701b88c54edcb" exitCode=60 Nov 01 22:45:05.003356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.003275 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" event={"ID":"6fc6ede6-5a8f-400a-8557-287c62afcc01","Type":"ContainerDied","Data":"d10d696009be5de7b8a91614971c3ee477d7e4b056b267abc02701b88c54edcb"} Nov 01 22:45:05.003418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.003344 4981 scope.go:122] "RemoveContainer" containerID="7be8aeba0d3143d4f6e7c86c5c7fb683f6d1050adf7e6936163fb7a0765b432f" Nov 01 22:45:05.003613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.003596 4981 scope.go:122] "RemoveContainer" containerID="d10d696009be5de7b8a91614971c3ee477d7e4b056b267abc02701b88c54edcb" Nov 01 22:45:05.004035 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:05.004011 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq_odh-ai-gateway-infra(6fc6ede6-5a8f-400a-8557-287c62afcc01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" Nov 01 22:45:05.005360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.005336 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerStarted","Data":"5b265869b28764628334c06ca8462c9ae2687dd26d06e685bea98e453017afdf"} Nov 01 22:45:05.007497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.007474 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerStarted","Data":"00fac7b078bd4803b7f637c37d8dcfb6f593fb15089953a5cb699be816b6afd2"} Nov 01 22:45:05.009750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.009718 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerStarted","Data":"f1cee820280518ffab123590ca23c262a278200367074ed6e27de20efbcbecfb"} Nov 01 22:45:05.013284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.013259 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" event={"ID":"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3","Type":"ContainerStarted","Data":"a040f965eb78637bf5b36bca380eccf6f808ef8e578aab699b7338569a8255ef"} Nov 01 22:45:05.016546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.016521 4981 generic.go:356] "Generic (PLEG): container finished" podID="ce1efe76-4df3-492e-be84-4c58ad778677" containerID="2f6be5f9a4bbd83aaadb68a082787df50816404a40e827109a2e20570ba8e8a0" exitCode=60 Nov 01 22:45:05.016708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.016689 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" event={"ID":"ce1efe76-4df3-492e-be84-4c58ad778677","Type":"ContainerDied","Data":"2f6be5f9a4bbd83aaadb68a082787df50816404a40e827109a2e20570ba8e8a0"} Nov 01 22:45:05.017136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.017114 4981 scope.go:122] "RemoveContainer" containerID="2f6be5f9a4bbd83aaadb68a082787df50816404a40e827109a2e20570ba8e8a0" Nov 01 22:45:05.017535 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:05.017510 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp_odh-ai-gateway-infra(ce1efe76-4df3-492e-be84-4c58ad778677)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" Nov 01 22:45:05.023046 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.023020 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerStarted","Data":"a44951bade595c1d1e4d0cfa3979e234716b47dd4217a7cee6135618f557906e"} Nov 01 22:45:05.025437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.025399 4981 scope.go:122] "RemoveContainer" containerID="7297a7753121b54c4cc7897cd82867701bf68a1a33f9b23db12a3472ae2457a5" Nov 01 22:45:05.031053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.031027 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerStarted","Data":"eee41ec7549390a4dbda9a15a8a743f96023dc27d5d6d1aa9fe9ba53cad89e57"} Nov 01 22:45:05.033677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.033654 4981 generic.go:356] "Generic (PLEG): container finished" podID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerID="ce99e929752c1e7d4315f266c122be5b9c7714f2983ea24efbd9ce9aa771b435" exitCode=60 Nov 01 22:45:05.033772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.033744 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" event={"ID":"cd3eb503-df56-4cad-ad04-91b08c7f0f6b","Type":"ContainerDied","Data":"ce99e929752c1e7d4315f266c122be5b9c7714f2983ea24efbd9ce9aa771b435"} Nov 01 22:45:05.033935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.033920 4981 scope.go:122] "RemoveContainer" containerID="ce99e929752c1e7d4315f266c122be5b9c7714f2983ea24efbd9ce9aa771b435" Nov 01 22:45:05.034213 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:05.034195 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc_odh-ai-gateway-infra(cd3eb503-df56-4cad-ad04-91b08c7f0f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" Nov 01 22:45:05.044141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.044082 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n"] Nov 01 22:45:05.045637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.045501 4981 scope.go:122] "RemoveContainer" containerID="ea648bf767c633557a377e3b46e0271e57afad461380bed530141c3c35377c0e" Nov 01 22:45:05.053666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.053640 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j"] Nov 01 22:45:05.061244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.061214 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38563234-6ms2n"] Nov 01 22:45:05.079306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.073945 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad0fe7f-2v25j"] Nov 01 22:45:05.192964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.192928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:05.817035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.817002 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799"] Nov 01 22:45:05.818296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.818267 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2754c0df-eb00-4688-b7eb-112aa189c0fa" containerName="revoke-keys" Nov 01 22:45:05.818296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.818290 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2754c0df-eb00-4688-b7eb-112aa189c0fa" containerName="revoke-keys" Nov 01 22:45:05.818508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.818438 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2754c0df-eb00-4688-b7eb-112aa189c0fa" containerName="revoke-keys" Nov 01 22:45:05.822759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.822738 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:05.832945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.832914 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:45:05.844623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.844552 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799"] Nov 01 22:45:05.971834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.971787 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75rk\" (UniqueName: \"kubernetes.io/projected/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-kube-api-access-v75rk\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:05.972090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.972070 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:05.972239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:05.972217 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.046530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.046447 4981 scope.go:122] "RemoveContainer" containerID="d10d696009be5de7b8a91614971c3ee477d7e4b056b267abc02701b88c54edcb" Nov 01 22:45:06.046996 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.046969 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq_odh-ai-gateway-infra(6fc6ede6-5a8f-400a-8557-287c62afcc01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" Nov 01 22:45:06.048258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.048237 4981 generic.go:356] "Generic (PLEG): container finished" podID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerID="5b265869b28764628334c06ca8462c9ae2687dd26d06e685bea98e453017afdf" exitCode=60 Nov 01 22:45:06.048357 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.048318 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerDied","Data":"5b265869b28764628334c06ca8462c9ae2687dd26d06e685bea98e453017afdf"} Nov 01 22:45:06.048408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.048384 4981 scope.go:122] "RemoveContainer" containerID="0cd273544de30ab0ca3d21e7db2af5500200c12a1f9d447eef245afa5df6adce" Nov 01 22:45:06.048670 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.048650 4981 scope.go:122] "RemoveContainer" containerID="5b265869b28764628334c06ca8462c9ae2687dd26d06e685bea98e453017afdf" Nov 01 22:45:06.049075 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.049050 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn_odh-ai-gateway-infra(b1224a3d-870d-4c91-8508-ad97ad1cf806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" Nov 01 22:45:06.050351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.050331 4981 generic.go:356] "Generic (PLEG): container finished" podID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerID="00fac7b078bd4803b7f637c37d8dcfb6f593fb15089953a5cb699be816b6afd2" exitCode=60 Nov 01 22:45:06.050437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.050416 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerDied","Data":"00fac7b078bd4803b7f637c37d8dcfb6f593fb15089953a5cb699be816b6afd2"} Nov 01 22:45:06.050626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.050610 4981 scope.go:122] "RemoveContainer" containerID="00fac7b078bd4803b7f637c37d8dcfb6f593fb15089953a5cb699be816b6afd2" Nov 01 22:45:06.050913 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.050893 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4_odh-ai-gateway-infra(eb47cc2a-e221-44bb-946c-7eca7267a452)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" Nov 01 22:45:06.052400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.052382 4981 generic.go:356] "Generic (PLEG): container finished" podID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerID="f1cee820280518ffab123590ca23c262a278200367074ed6e27de20efbcbecfb" exitCode=60 Nov 01 22:45:06.052487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.052464 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerDied","Data":"f1cee820280518ffab123590ca23c262a278200367074ed6e27de20efbcbecfb"} Nov 01 22:45:06.052660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.052644 4981 scope.go:122] "RemoveContainer" containerID="f1cee820280518ffab123590ca23c262a278200367074ed6e27de20efbcbecfb" Nov 01 22:45:06.052905 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.052884 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx_odh-ai-gateway-infra(05dfc6d0-c041-47aa-86e1-227e3b32843b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" Nov 01 22:45:06.054404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.054384 4981 generic.go:356] "Generic (PLEG): container finished" podID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerID="a040f965eb78637bf5b36bca380eccf6f808ef8e578aab699b7338569a8255ef" exitCode=60 Nov 01 22:45:06.054493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.054462 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" event={"ID":"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3","Type":"ContainerDied","Data":"a040f965eb78637bf5b36bca380eccf6f808ef8e578aab699b7338569a8255ef"} Nov 01 22:45:06.054692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.054674 4981 scope.go:122] "RemoveContainer" containerID="a040f965eb78637bf5b36bca380eccf6f808ef8e578aab699b7338569a8255ef" Nov 01 22:45:06.054969 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.054943 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz_odh-ai-gateway-infra(2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" Nov 01 22:45:06.056405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.056386 4981 scope.go:122] "RemoveContainer" containerID="2f6be5f9a4bbd83aaadb68a082787df50816404a40e827109a2e20570ba8e8a0" Nov 01 22:45:06.056678 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.056656 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp_odh-ai-gateway-infra(ce1efe76-4df3-492e-be84-4c58ad778677)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" Nov 01 22:45:06.058400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.058377 4981 generic.go:356] "Generic (PLEG): container finished" podID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerID="a44951bade595c1d1e4d0cfa3979e234716b47dd4217a7cee6135618f557906e" exitCode=60 Nov 01 22:45:06.058488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.058457 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerDied","Data":"a44951bade595c1d1e4d0cfa3979e234716b47dd4217a7cee6135618f557906e"} Nov 01 22:45:06.058661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.058645 4981 scope.go:122] "RemoveContainer" containerID="a44951bade595c1d1e4d0cfa3979e234716b47dd4217a7cee6135618f557906e" Nov 01 22:45:06.058884 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.058861 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg_odh-ai-gateway-infra(8472535c-7063-4bea-82b9-39a6ee919bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" Nov 01 22:45:06.060395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.060375 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerID="eb93c94dc49b700c4001617836e9dfbf6e260a16ca8f70c9970cd98d03603fd6" exitCode=60 Nov 01 22:45:06.060490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.060443 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" event={"ID":"dc92015e-f197-4c25-ba89-fafc04e006b8","Type":"ContainerDied","Data":"eb93c94dc49b700c4001617836e9dfbf6e260a16ca8f70c9970cd98d03603fd6"} Nov 01 22:45:06.060771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.060750 4981 scope.go:122] "RemoveContainer" containerID="eb93c94dc49b700c4001617836e9dfbf6e260a16ca8f70c9970cd98d03603fd6" Nov 01 22:45:06.061062 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.061026 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp_odh-ai-gateway-infra(dc92015e-f197-4c25-ba89-fafc04e006b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" Nov 01 22:45:06.063181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.063155 4981 generic.go:356] "Generic (PLEG): container finished" podID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerID="eee41ec7549390a4dbda9a15a8a743f96023dc27d5d6d1aa9fe9ba53cad89e57" exitCode=60 Nov 01 22:45:06.063294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.063230 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerDied","Data":"eee41ec7549390a4dbda9a15a8a743f96023dc27d5d6d1aa9fe9ba53cad89e57"} Nov 01 22:45:06.063425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.063409 4981 scope.go:122] "RemoveContainer" containerID="eee41ec7549390a4dbda9a15a8a743f96023dc27d5d6d1aa9fe9ba53cad89e57" Nov 01 22:45:06.063690 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.063666 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4_odh-ai-gateway-infra(4f36e99e-f6ff-4007-a2ca-61792b5982bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" Nov 01 22:45:06.065126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.065108 4981 scope.go:122] "RemoveContainer" containerID="ce99e929752c1e7d4315f266c122be5b9c7714f2983ea24efbd9ce9aa771b435" Nov 01 22:45:06.065375 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.065352 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc_odh-ai-gateway-infra(cd3eb503-df56-4cad-ad04-91b08c7f0f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" Nov 01 22:45:06.066016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.066001 4981 scope.go:122] "RemoveContainer" containerID="8f4e5b13adb8e05837e3a96ebdc8198db3a8acf9a9a13640b668e06a80c9cd53" Nov 01 22:45:06.066889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.066869 4981 generic.go:356] "Generic (PLEG): container finished" podID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerID="46f9afbce6c0aebd26bb5971e4b3166c6d55090e65b93ab6172e51f8a803f276" exitCode=60 Nov 01 22:45:06.066995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.066950 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" event={"ID":"6c40734f-e0ed-49ef-b552-8e29e6904fa3","Type":"ContainerDied","Data":"46f9afbce6c0aebd26bb5971e4b3166c6d55090e65b93ab6172e51f8a803f276"} Nov 01 22:45:06.067114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.067100 4981 scope.go:122] "RemoveContainer" containerID="46f9afbce6c0aebd26bb5971e4b3166c6d55090e65b93ab6172e51f8a803f276" Nov 01 22:45:06.067318 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:06.067301 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd_odh-ai-gateway-infra(6c40734f-e0ed-49ef-b552-8e29e6904fa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" Nov 01 22:45:06.073617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.073553 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.073867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.073846 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75rk\" (UniqueName: \"kubernetes.io/projected/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-kube-api-access-v75rk\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.074043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.074024 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.074257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.074238 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.076819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.076478 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.081740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.081716 4981 scope.go:122] "RemoveContainer" containerID="63457c08cc556c598870b145a8b924118d6882d20debdb25f27cbe7b82dd9ce1" Nov 01 22:45:06.086886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.084811 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75rk\" (UniqueName: \"kubernetes.io/projected/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-kube-api-access-v75rk\") pod \"maas-api-revoke-keys-e2e-mt-30578128-l9799\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.119766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.119734 4981 scope.go:122] "RemoveContainer" containerID="08770bd1f5da5fa43a8527d3953fd24900c3e21f9bf58a8da290f4bd61d4a762" Nov 01 22:45:06.138060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.138036 4981 scope.go:122] "RemoveContainer" containerID="834fa2af44a3cb1036162b66a0f6330cfd4ec6a8d0537c7837e2409fdf28905c" Nov 01 22:45:06.161144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.157826 4981 scope.go:122] "RemoveContainer" containerID="1543fc6db5aad1dce402f347cd03b986a5b12f904f43f280fed1e9dba17a8322" Nov 01 22:45:06.169618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.169550 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:06.175668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.175598 4981 scope.go:122] "RemoveContainer" containerID="53baeaefc4348c97da46a2ba7f37cea739d5b7a1685fcda90c79e70e2de34dc2" Nov 01 22:45:06.265242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.265208 4981 scope.go:122] "RemoveContainer" containerID="9c320d725a9bcf76d590d89adfebb306e3a7e959b39d2fda8fc39b2b055ca041" Nov 01 22:45:06.408023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.407932 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:06.511962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.511933 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799"] Nov 01 22:45:06.513444 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:06.513421 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f83d7b3_90e6_4e93_80a4_1f7c7a41bc9c.slice/crio-4a891733b8ee0c67f36e40f1cd646ca0f40107923705cc24919fdf66634bfc4c WatchSource:0}: Error finding container 4a891733b8ee0c67f36e40f1cd646ca0f40107923705cc24919fdf66634bfc4c: Status 404 returned error can't find the container with id 4a891733b8ee0c67f36e40f1cd646ca0f40107923705cc24919fdf66634bfc4c Nov 01 22:45:06.748878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.748825 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7" path="/var/lib/kubelet/pods/50e53eb8-52ca-49dd-b0ad-0dd1f5da30b7/volumes" Nov 01 22:45:06.750059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:06.750031 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13b87a9-110f-44ac-9f70-a7348c6723df" path="/var/lib/kubelet/pods/c13b87a9-110f-44ac-9f70-a7348c6723df/volumes" Nov 01 22:45:07.079332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.079239 4981 scope.go:122] "RemoveContainer" containerID="eee41ec7549390a4dbda9a15a8a743f96023dc27d5d6d1aa9fe9ba53cad89e57" Nov 01 22:45:07.079812 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:07.079759 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4_odh-ai-gateway-infra(4f36e99e-f6ff-4007-a2ca-61792b5982bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" Nov 01 22:45:07.081087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.081056 4981 scope.go:122] "RemoveContainer" containerID="46f9afbce6c0aebd26bb5971e4b3166c6d55090e65b93ab6172e51f8a803f276" Nov 01 22:45:07.081527 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:07.081499 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd_odh-ai-gateway-infra(6c40734f-e0ed-49ef-b552-8e29e6904fa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" Nov 01 22:45:07.082867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.082849 4981 scope.go:122] "RemoveContainer" containerID="5b265869b28764628334c06ca8462c9ae2687dd26d06e685bea98e453017afdf" Nov 01 22:45:07.083210 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:07.083190 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn_odh-ai-gateway-infra(b1224a3d-870d-4c91-8508-ad97ad1cf806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" Nov 01 22:45:07.084432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.084414 4981 scope.go:122] "RemoveContainer" containerID="00fac7b078bd4803b7f637c37d8dcfb6f593fb15089953a5cb699be816b6afd2" Nov 01 22:45:07.084776 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:07.084752 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4_odh-ai-gateway-infra(eb47cc2a-e221-44bb-946c-7eca7267a452)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" Nov 01 22:45:07.086157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.086134 4981 scope.go:122] "RemoveContainer" containerID="f1cee820280518ffab123590ca23c262a278200367074ed6e27de20efbcbecfb" Nov 01 22:45:07.086481 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:07.086455 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx_odh-ai-gateway-infra(05dfc6d0-c041-47aa-86e1-227e3b32843b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" Nov 01 22:45:07.087479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.087457 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" event={"ID":"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c","Type":"ContainerStarted","Data":"dbdbf6ccc345a87c35407fa706b86fbcefb80496b1f7960738ba7636b8516517"} Nov 01 22:45:07.087596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.087517 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" event={"ID":"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c","Type":"ContainerStarted","Data":"4a891733b8ee0c67f36e40f1cd646ca0f40107923705cc24919fdf66634bfc4c"} Nov 01 22:45:07.089064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.089044 4981 generic.go:356] "Generic (PLEG): container finished" podID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerID="10f7f23bf0a0ecbc00644104539551629839257ee93574c9504540714f59cb7f" exitCode=0 Nov 01 22:45:07.089152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.089111 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" event={"ID":"1bef9cd2-8fd6-42e4-865c-0004f50dd88a","Type":"ContainerDied","Data":"10f7f23bf0a0ecbc00644104539551629839257ee93574c9504540714f59cb7f"} Nov 01 22:45:07.089436 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.089420 4981 scope.go:122] "RemoveContainer" containerID="10f7f23bf0a0ecbc00644104539551629839257ee93574c9504540714f59cb7f" Nov 01 22:45:07.091082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.091058 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:07.091162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.091117 4981 scope.go:122] "RemoveContainer" containerID="a040f965eb78637bf5b36bca380eccf6f808ef8e578aab699b7338569a8255ef" Nov 01 22:45:07.091162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.091135 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:07.091247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.091172 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:07.091398 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:07.091378 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz_odh-ai-gateway-infra(2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" Nov 01 22:45:07.094405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.094387 4981 scope.go:122] "RemoveContainer" containerID="a44951bade595c1d1e4d0cfa3979e234716b47dd4217a7cee6135618f557906e" Nov 01 22:45:07.094710 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:07.094668 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg_odh-ai-gateway-infra(8472535c-7063-4bea-82b9-39a6ee919bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" Nov 01 22:45:07.322470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.322375 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:07.342206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:07.342086 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" podStartSLOduration=2.342063627 podStartE2EDuration="2.342063627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:07.307372374 +0000 UTC m=+1272.735250307" watchObservedRunningTime="2026-11-01 22:45:07.342063627 +0000 UTC m=+1272.769941562" Nov 01 22:45:08.057631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.057553 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7"] Nov 01 22:45:08.064250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.064219 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0a3c6ae-bv4s7"] Nov 01 22:45:08.108554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.108519 4981 generic.go:356] "Generic (PLEG): container finished" podID="7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" containerID="dbdbf6ccc345a87c35407fa706b86fbcefb80496b1f7960738ba7636b8516517" exitCode=0 Nov 01 22:45:08.108983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.108616 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" event={"ID":"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c","Type":"ContainerDied","Data":"dbdbf6ccc345a87c35407fa706b86fbcefb80496b1f7960738ba7636b8516517"} Nov 01 22:45:08.110941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.110909 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" event={"ID":"1bef9cd2-8fd6-42e4-865c-0004f50dd88a","Type":"ContainerStarted","Data":"7e5a1789f54c21eb401ae96d9c50c48296bb04e88102e949241286082c6f762a"} Nov 01 22:45:08.216122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.216075 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:08.260518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.260484 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:45:08.260822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.260795 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:45:08.260923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.260848 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 22:45:08.740039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:08.740004 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c509abba-f1b5-4442-8e6d-c762fce969d0" path="/var/lib/kubelet/pods/c509abba-f1b5-4442-8e6d-c762fce969d0/volumes" Nov 01 22:45:09.031367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.031284 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt"] Nov 01 22:45:09.036798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.036770 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b3d6ecc-txzgt"] Nov 01 22:45:09.208043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.207999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:09.348469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.345282 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:45:09.348469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.345344 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 22:45:09.352402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.352362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:09.456164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.453051 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:09.469704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.469679 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:09.512530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.511902 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-xks-ca\") pod \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " Nov 01 22:45:09.512530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.511979 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-openshift-ca\") pod \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " Nov 01 22:45:09.512530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.512171 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-kube-api-access-v75rk\" (UniqueName: \"kubernetes.io/projected/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-kube-api-access-v75rk\") pod \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\" (UID: \"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c\") " Nov 01 22:45:09.512874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.512616 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-openshift-ca" pod "7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" (UID: "7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:09.513116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.513095 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:09.514409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.514380 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-xks-ca" pod "7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" (UID: "7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:09.515253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.515224 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-kube-api-access-v75rk" pod "7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" (UID: "7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c"). InnerVolumeSpecName "kube-api-access-v75rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:09.615179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.615099 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:09.615179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:09.615130 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75rk\" (UniqueName: \"kubernetes.io/projected/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c-kube-api-access-v75rk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:10.267597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:10.267550 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:45:10.268109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:10.267627 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 22:45:10.362776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:10.329956 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" Nov 01 22:45:10.362776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:10.331914 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799" event={"ID":"7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c","Type":"ContainerDied","Data":"4a891733b8ee0c67f36e40f1cd646ca0f40107923705cc24919fdf66634bfc4c"} Nov 01 22:45:10.362776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:10.332078 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a891733b8ee0c67f36e40f1cd646ca0f40107923705cc24919fdf66634bfc4c" Nov 01 22:45:10.741411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:10.741379 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c49ffc-13c2-413e-8618-b335f3a49890" path="/var/lib/kubelet/pods/d0c49ffc-13c2-413e-8618-b335f3a49890/volumes" Nov 01 22:45:10.761203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:10.761171 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:11.103861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:11.103771 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92"] Nov 01 22:45:11.109398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:11.109366 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e34bc39a-cxf92"] Nov 01 22:45:11.260128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:11.260090 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:45:11.260308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:11.260188 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 22:45:11.773779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:11.773746 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:11.992218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:11.992076 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:12.259985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:12.259942 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:45:12.260164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:12.260004 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 22:45:12.704982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:12.704943 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:12.848861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:12.848828 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d97c14-60db-45b5-ac17-13d5e9a2293e" path="/var/lib/kubelet/pods/b3d97c14-60db-45b5-ac17-13d5e9a2293e/volumes" Nov 01 22:45:13.260322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:13.260287 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:45:13.260504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:13.260345 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 22:45:13.563308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:13.563229 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:13.708905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:13.707569 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:14.260360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:14.260322 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 01 22:45:14.260978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:14.260942 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 01 22:45:14.437793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:14.437754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:14.540884 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:14.540799 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:45:15.241304 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.241276 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:15.265689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.265563 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:45:15.266180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.266103 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:45:15.267090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.267063 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:45:15.727194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.727163 4981 scope.go:122] "RemoveContainer" containerID="ce99e929752c1e7d4315f266c122be5b9c7714f2983ea24efbd9ce9aa771b435" Nov 01 22:45:15.900986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.900950 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5"] Nov 01 22:45:15.913235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.913208 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z"] Nov 01 22:45:15.968221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:15.967995 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:16.083525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:16.083494 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:16.235598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:16.235482 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:16.474595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:16.465184 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" event={"ID":"cd3eb503-df56-4cad-ad04-91b08c7f0f6b","Type":"ContainerStarted","Data":"06be32caea996df5d2fd402a9f1926757a305c3192524540c6c3572469bd7724"} Nov 01 22:45:16.726927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:16.726834 4981 scope.go:122] "RemoveContainer" containerID="eee41ec7549390a4dbda9a15a8a743f96023dc27d5d6d1aa9fe9ba53cad89e57" Nov 01 22:45:16.727108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:16.727077 4981 scope.go:122] "RemoveContainer" containerID="eb93c94dc49b700c4001617836e9dfbf6e260a16ca8f70c9970cd98d03603fd6" Nov 01 22:45:16.730593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:16.727458 4981 scope.go:122] "RemoveContainer" containerID="5b265869b28764628334c06ca8462c9ae2687dd26d06e685bea98e453017afdf" Nov 01 22:45:16.730593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:16.727991 4981 scope.go:122] "RemoveContainer" containerID="d10d696009be5de7b8a91614971c3ee477d7e4b056b267abc02701b88c54edcb" Nov 01 22:45:17.055430 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.055396 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:17.479051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.479016 4981 generic.go:356] "Generic (PLEG): container finished" podID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerID="06be32caea996df5d2fd402a9f1926757a305c3192524540c6c3572469bd7724" exitCode=60 Nov 01 22:45:17.479531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.479106 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" event={"ID":"cd3eb503-df56-4cad-ad04-91b08c7f0f6b","Type":"ContainerDied","Data":"06be32caea996df5d2fd402a9f1926757a305c3192524540c6c3572469bd7724"} Nov 01 22:45:17.479531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.479179 4981 scope.go:122] "RemoveContainer" containerID="ce99e929752c1e7d4315f266c122be5b9c7714f2983ea24efbd9ce9aa771b435" Nov 01 22:45:17.479734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.479717 4981 scope.go:122] "RemoveContainer" containerID="06be32caea996df5d2fd402a9f1926757a305c3192524540c6c3572469bd7724" Nov 01 22:45:17.480212 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:17.480185 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc_odh-ai-gateway-infra(cd3eb503-df56-4cad-ad04-91b08c7f0f6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" Nov 01 22:45:17.481738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.481716 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" event={"ID":"6fc6ede6-5a8f-400a-8557-287c62afcc01","Type":"ContainerStarted","Data":"963ade6326d4b1ee6e0ae4186e618073f654e83e705b6cc807f422e1362c7753"} Nov 01 22:45:17.483941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.483920 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerStarted","Data":"977bd1558d05c2d79bcf633b427134d88bd4399b57988a3a49e597ecc139d67d"} Nov 01 22:45:17.486327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.486304 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" event={"ID":"dc92015e-f197-4c25-ba89-fafc04e006b8","Type":"ContainerStarted","Data":"078d45bae96a66fc1df42836b9703ddead0faedfd62a13064dc86813bb963b39"} Nov 01 22:45:17.488373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.488350 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerStarted","Data":"67f4ce2f1681065f1489555b49ed530febf412d061284a7464cae30f81af0518"} Nov 01 22:45:17.512710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.512682 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc"] Nov 01 22:45:17.727535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.727501 4981 scope.go:122] "RemoveContainer" containerID="00fac7b078bd4803b7f637c37d8dcfb6f593fb15089953a5cb699be816b6afd2" Nov 01 22:45:17.728634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.728610 4981 scope.go:122] "RemoveContainer" containerID="a44951bade595c1d1e4d0cfa3979e234716b47dd4217a7cee6135618f557906e" Nov 01 22:45:17.729869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.729842 4981 scope.go:122] "RemoveContainer" containerID="2f6be5f9a4bbd83aaadb68a082787df50816404a40e827109a2e20570ba8e8a0" Nov 01 22:45:17.730337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.730311 4981 scope.go:122] "RemoveContainer" containerID="f1cee820280518ffab123590ca23c262a278200367074ed6e27de20efbcbecfb" Nov 01 22:45:17.773292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.773259 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp"] Nov 01 22:45:17.895226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:17.895195 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:18.501204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.501171 4981 generic.go:356] "Generic (PLEG): container finished" podID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerID="963ade6326d4b1ee6e0ae4186e618073f654e83e705b6cc807f422e1362c7753" exitCode=60 Nov 01 22:45:18.501750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.501251 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" event={"ID":"6fc6ede6-5a8f-400a-8557-287c62afcc01","Type":"ContainerDied","Data":"963ade6326d4b1ee6e0ae4186e618073f654e83e705b6cc807f422e1362c7753"} Nov 01 22:45:18.501750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.501350 4981 scope.go:122] "RemoveContainer" containerID="d10d696009be5de7b8a91614971c3ee477d7e4b056b267abc02701b88c54edcb" Nov 01 22:45:18.501941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.501922 4981 scope.go:122] "RemoveContainer" containerID="963ade6326d4b1ee6e0ae4186e618073f654e83e705b6cc807f422e1362c7753" Nov 01 22:45:18.502431 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:18.502387 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq_odh-ai-gateway-infra(6fc6ede6-5a8f-400a-8557-287c62afcc01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" Nov 01 22:45:18.503605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.503585 4981 generic.go:356] "Generic (PLEG): container finished" podID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerID="977bd1558d05c2d79bcf633b427134d88bd4399b57988a3a49e597ecc139d67d" exitCode=60 Nov 01 22:45:18.503706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.503622 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerDied","Data":"977bd1558d05c2d79bcf633b427134d88bd4399b57988a3a49e597ecc139d67d"} Nov 01 22:45:18.504052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.504035 4981 scope.go:122] "RemoveContainer" containerID="977bd1558d05c2d79bcf633b427134d88bd4399b57988a3a49e597ecc139d67d" Nov 01 22:45:18.504445 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:18.504425 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn_odh-ai-gateway-infra(b1224a3d-870d-4c91-8508-ad97ad1cf806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" Nov 01 22:45:18.505532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.505511 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerStarted","Data":"c9f4cfa652b6e2763cc647f73e3687d0e55c072a6cf60360ad155a152c83c9f2"} Nov 01 22:45:18.507625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.507604 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerStarted","Data":"fb1b7762463e03d185fb945e485e39b1d411276c73fbe213c6f5aec6cca43cdb"} Nov 01 22:45:18.509619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.509596 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" event={"ID":"ce1efe76-4df3-492e-be84-4c58ad778677","Type":"ContainerStarted","Data":"a642c55a97b39942e173d08d4e788e14453ad10d023be9031c8fc66ef90ca271"} Nov 01 22:45:18.511610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.511568 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerStarted","Data":"b82147bc1be13ea1648722b2d9fa40d862af148134c33dbc391b6c61049c882a"} Nov 01 22:45:18.513410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.513389 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerID="078d45bae96a66fc1df42836b9703ddead0faedfd62a13064dc86813bb963b39" exitCode=60 Nov 01 22:45:18.513500 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.513472 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" event={"ID":"dc92015e-f197-4c25-ba89-fafc04e006b8","Type":"ContainerDied","Data":"078d45bae96a66fc1df42836b9703ddead0faedfd62a13064dc86813bb963b39"} Nov 01 22:45:18.515670 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.515648 4981 generic.go:356] "Generic (PLEG): container finished" podID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerID="67f4ce2f1681065f1489555b49ed530febf412d061284a7464cae30f81af0518" exitCode=60 Nov 01 22:45:18.515776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.515730 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerDied","Data":"67f4ce2f1681065f1489555b49ed530febf412d061284a7464cae30f81af0518"} Nov 01 22:45:18.516119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.516094 4981 scope.go:122] "RemoveContainer" containerID="67f4ce2f1681065f1489555b49ed530febf412d061284a7464cae30f81af0518" Nov 01 22:45:18.516464 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:18.516435 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4_odh-ai-gateway-infra(4f36e99e-f6ff-4007-a2ca-61792b5982bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" Nov 01 22:45:18.518489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.518470 4981 scope.go:122] "RemoveContainer" containerID="5b265869b28764628334c06ca8462c9ae2687dd26d06e685bea98e453017afdf" Nov 01 22:45:18.534534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.534512 4981 scope.go:122] "RemoveContainer" containerID="eb93c94dc49b700c4001617836e9dfbf6e260a16ca8f70c9970cd98d03603fd6" Nov 01 22:45:18.544065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.543861 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn"] Nov 01 22:45:18.566302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.566280 4981 scope.go:122] "RemoveContainer" containerID="eee41ec7549390a4dbda9a15a8a743f96023dc27d5d6d1aa9fe9ba53cad89e57" Nov 01 22:45:18.636607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.635862 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4"] Nov 01 22:45:18.648683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.646882 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq"] Nov 01 22:45:18.737751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.735158 4981 scope.go:122] "RemoveContainer" containerID="46f9afbce6c0aebd26bb5971e4b3166c6d55090e65b93ab6172e51f8a803f276" Nov 01 22:45:18.741730 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.741667 4981 scope.go:122] "RemoveContainer" containerID="a040f965eb78637bf5b36bca380eccf6f808ef8e578aab699b7338569a8255ef" Nov 01 22:45:18.800651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.797810 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:18.800651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.800007 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:18.936643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.936609 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc92015e-f197-4c25-ba89-fafc04e006b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc92015e-f197-4c25-ba89-fafc04e006b8-maas-api-service-ca\") pod \"dc92015e-f197-4c25-ba89-fafc04e006b8\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " Nov 01 22:45:18.936812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.936716 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-maas-api-service-ca\") pod \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " Nov 01 22:45:18.936812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.936791 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-kube-api-access-4n6wz\" (UniqueName: \"kubernetes.io/projected/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-kube-api-access-4n6wz\") pod \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\" (UID: \"cd3eb503-df56-4cad-ad04-91b08c7f0f6b\") " Nov 01 22:45:18.937034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.937007 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc92015e-f197-4c25-ba89-fafc04e006b8-maas-api-service-ca" pod "dc92015e-f197-4c25-ba89-fafc04e006b8" (UID: "dc92015e-f197-4c25-ba89-fafc04e006b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:18.937121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.937033 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-maas-api-service-ca" pod "cd3eb503-df56-4cad-ad04-91b08c7f0f6b" (UID: "cd3eb503-df56-4cad-ad04-91b08c7f0f6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:18.937121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.937020 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc92015e-f197-4c25-ba89-fafc04e006b8-kube-api-access-mkjtb\" (UniqueName: \"kubernetes.io/projected/dc92015e-f197-4c25-ba89-fafc04e006b8-kube-api-access-mkjtb\") pod \"dc92015e-f197-4c25-ba89-fafc04e006b8\" (UID: \"dc92015e-f197-4c25-ba89-fafc04e006b8\") " Nov 01 22:45:18.937819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.937790 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc92015e-f197-4c25-ba89-fafc04e006b8-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:18.937819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.937818 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:18.981852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.981816 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc92015e-f197-4c25-ba89-fafc04e006b8-kube-api-access-mkjtb" pod "dc92015e-f197-4c25-ba89-fafc04e006b8" (UID: "dc92015e-f197-4c25-ba89-fafc04e006b8"). InnerVolumeSpecName "kube-api-access-mkjtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:18.985048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:18.982736 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-kube-api-access-4n6wz" pod "cd3eb503-df56-4cad-ad04-91b08c7f0f6b" (UID: "cd3eb503-df56-4cad-ad04-91b08c7f0f6b"). InnerVolumeSpecName "kube-api-access-4n6wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:19.040543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.040507 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6wz\" (UniqueName: \"kubernetes.io/projected/cd3eb503-df56-4cad-ad04-91b08c7f0f6b-kube-api-access-4n6wz\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:19.040719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.040546 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkjtb\" (UniqueName: \"kubernetes.io/projected/dc92015e-f197-4c25-ba89-fafc04e006b8-kube-api-access-mkjtb\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:19.202085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.202048 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx"] Nov 01 22:45:19.205940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.205914 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:19.576614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.572303 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:19.641597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.640982 4981 generic.go:356] "Generic (PLEG): container finished" podID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerID="c9f4cfa652b6e2763cc647f73e3687d0e55c072a6cf60360ad155a152c83c9f2" exitCode=60 Nov 01 22:45:19.641597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.641104 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerDied","Data":"c9f4cfa652b6e2763cc647f73e3687d0e55c072a6cf60360ad155a152c83c9f2"} Nov 01 22:45:19.641597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.641189 4981 scope.go:122] "RemoveContainer" containerID="00fac7b078bd4803b7f637c37d8dcfb6f593fb15089953a5cb699be816b6afd2" Nov 01 22:45:19.642598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.642021 4981 scope.go:122] "RemoveContainer" containerID="c9f4cfa652b6e2763cc647f73e3687d0e55c072a6cf60360ad155a152c83c9f2" Nov 01 22:45:19.642598 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:19.642504 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4_odh-ai-gateway-infra(eb47cc2a-e221-44bb-946c-7eca7267a452)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" Nov 01 22:45:19.646531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.646508 4981 generic.go:356] "Generic (PLEG): container finished" podID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerID="fb1b7762463e03d185fb945e485e39b1d411276c73fbe213c6f5aec6cca43cdb" exitCode=60 Nov 01 22:45:19.646963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.646933 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerDied","Data":"fb1b7762463e03d185fb945e485e39b1d411276c73fbe213c6f5aec6cca43cdb"} Nov 01 22:45:19.670810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.670778 4981 generic.go:356] "Generic (PLEG): container finished" podID="ce1efe76-4df3-492e-be84-4c58ad778677" containerID="a642c55a97b39942e173d08d4e788e14453ad10d023be9031c8fc66ef90ca271" exitCode=60 Nov 01 22:45:19.671113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.671092 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" event={"ID":"ce1efe76-4df3-492e-be84-4c58ad778677","Type":"ContainerDied","Data":"a642c55a97b39942e173d08d4e788e14453ad10d023be9031c8fc66ef90ca271"} Nov 01 22:45:19.675953 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.675933 4981 scope.go:122] "RemoveContainer" containerID="a642c55a97b39942e173d08d4e788e14453ad10d023be9031c8fc66ef90ca271" Nov 01 22:45:19.677271 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:19.677246 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp_odh-ai-gateway-infra(ce1efe76-4df3-492e-be84-4c58ad778677)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" Nov 01 22:45:19.678475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.678452 4981 generic.go:356] "Generic (PLEG): container finished" podID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerID="b82147bc1be13ea1648722b2d9fa40d862af148134c33dbc391b6c61049c882a" exitCode=60 Nov 01 22:45:19.678893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.678872 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerDied","Data":"b82147bc1be13ea1648722b2d9fa40d862af148134c33dbc391b6c61049c882a"} Nov 01 22:45:19.679651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.679633 4981 scope.go:122] "RemoveContainer" containerID="b82147bc1be13ea1648722b2d9fa40d862af148134c33dbc391b6c61049c882a" Nov 01 22:45:19.682956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.682922 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" event={"ID":"dc92015e-f197-4c25-ba89-fafc04e006b8","Type":"ContainerDied","Data":"1ab08004fbf03181dbab44ecf5e0c192f66259d394a49861634068923122e983"} Nov 01 22:45:19.683129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.683097 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp" Nov 01 22:45:19.683304 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:19.683280 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg_odh-ai-gateway-infra(8472535c-7063-4bea-82b9-39a6ee919bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" Nov 01 22:45:19.689918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.689884 4981 scope.go:122] "RemoveContainer" containerID="f1cee820280518ffab123590ca23c262a278200367074ed6e27de20efbcbecfb" Nov 01 22:45:19.712848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.710514 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" event={"ID":"cd3eb503-df56-4cad-ad04-91b08c7f0f6b","Type":"ContainerDied","Data":"64acb50598114fce0d514cdd69b35be933c68203a0b5d120cbada58797d1f314"} Nov 01 22:45:19.712848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.710777 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc" Nov 01 22:45:19.720272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.715667 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp"] Nov 01 22:45:19.803945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.803922 4981 scope.go:122] "RemoveContainer" containerID="2f6be5f9a4bbd83aaadb68a082787df50816404a40e827109a2e20570ba8e8a0" Nov 01 22:45:19.831486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.829963 4981 scope.go:122] "RemoveContainer" containerID="a44951bade595c1d1e4d0cfa3979e234716b47dd4217a7cee6135618f557906e" Nov 01 22:45:19.855563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.854876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:19.855563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.855107 4981 scope.go:122] "RemoveContainer" containerID="078d45bae96a66fc1df42836b9703ddead0faedfd62a13064dc86813bb963b39" Nov 01 22:45:19.866204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.864055 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4"] Nov 01 22:45:19.901435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.901011 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc"] Nov 01 22:45:19.901435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.901100 4981 scope.go:122] "RemoveContainer" containerID="06be32caea996df5d2fd402a9f1926757a305c3192524540c6c3572469bd7724" Nov 01 22:45:19.910706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.910684 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-dc635e-29892885-9wwnc"] Nov 01 22:45:19.933997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.921634 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7"] Nov 01 22:45:19.968468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.960341 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4534ad-pw5m7"] Nov 01 22:45:19.970145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.969670 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg"] Nov 01 22:45:19.999021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:19.997540 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp"] Nov 01 22:45:20.039040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.037396 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-7366bf-29892885-rvvdp"] Nov 01 22:45:20.073163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.073046 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:20.090883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.090509 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:20.105594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.102923 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc6ede6-5a8f-400a-8557-287c62afcc01-kube-api-access-4st8n\" (UniqueName: \"kubernetes.io/projected/6fc6ede6-5a8f-400a-8557-287c62afcc01-kube-api-access-4st8n\") pod \"6fc6ede6-5a8f-400a-8557-287c62afcc01\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " Nov 01 22:45:20.105594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.103012 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc6ede6-5a8f-400a-8557-287c62afcc01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6ede6-5a8f-400a-8557-287c62afcc01-maas-api-service-ca\") pod \"6fc6ede6-5a8f-400a-8557-287c62afcc01\" (UID: \"6fc6ede6-5a8f-400a-8557-287c62afcc01\") " Nov 01 22:45:20.112709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.112660 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc6ede6-5a8f-400a-8557-287c62afcc01-maas-api-service-ca" pod "6fc6ede6-5a8f-400a-8557-287c62afcc01" (UID: "6fc6ede6-5a8f-400a-8557-287c62afcc01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.131639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.121765 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc6ede6-5a8f-400a-8557-287c62afcc01-kube-api-access-4st8n" pod "6fc6ede6-5a8f-400a-8557-287c62afcc01" (UID: "6fc6ede6-5a8f-400a-8557-287c62afcc01"). InnerVolumeSpecName "kube-api-access-4st8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.207510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.206391 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4st8n\" (UniqueName: \"kubernetes.io/projected/6fc6ede6-5a8f-400a-8557-287c62afcc01-kube-api-access-4st8n\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.207510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.206420 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fc6ede6-5a8f-400a-8557-287c62afcc01-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.321719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.321691 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:20.326014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.325956 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:20.331269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.331248 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:20.381028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.380993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:20.409624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.409589 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f36e99e-f6ff-4007-a2ca-61792b5982bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36e99e-f6ff-4007-a2ca-61792b5982bd-maas-api-service-ca\") pod \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " Nov 01 22:45:20.409824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.409759 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1224a3d-870d-4c91-8508-ad97ad1cf806-kube-api-access-m24vv\" (UniqueName: \"kubernetes.io/projected/b1224a3d-870d-4c91-8508-ad97ad1cf806-kube-api-access-m24vv\") pod \"b1224a3d-870d-4c91-8508-ad97ad1cf806\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " Nov 01 22:45:20.409912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.409829 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1224a3d-870d-4c91-8508-ad97ad1cf806-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1224a3d-870d-4c91-8508-ad97ad1cf806-maas-api-service-ca\") pod \"b1224a3d-870d-4c91-8508-ad97ad1cf806\" (UID: \"b1224a3d-870d-4c91-8508-ad97ad1cf806\") " Nov 01 22:45:20.409976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.409913 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dfc6d0-c041-47aa-86e1-227e3b32843b-kube-api-access-2rw2s\" (UniqueName: \"kubernetes.io/projected/05dfc6d0-c041-47aa-86e1-227e3b32843b-kube-api-access-2rw2s\") pod \"05dfc6d0-c041-47aa-86e1-227e3b32843b\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " Nov 01 22:45:20.410123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.410108 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f36e99e-f6ff-4007-a2ca-61792b5982bd-kube-api-access-hrg2v\" (UniqueName: \"kubernetes.io/projected/4f36e99e-f6ff-4007-a2ca-61792b5982bd-kube-api-access-hrg2v\") pod \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\" (UID: \"4f36e99e-f6ff-4007-a2ca-61792b5982bd\") " Nov 01 22:45:20.410187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.410180 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05dfc6d0-c041-47aa-86e1-227e3b32843b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dfc6d0-c041-47aa-86e1-227e3b32843b-maas-api-service-ca\") pod \"05dfc6d0-c041-47aa-86e1-227e3b32843b\" (UID: \"05dfc6d0-c041-47aa-86e1-227e3b32843b\") " Nov 01 22:45:20.411403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.411276 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1224a3d-870d-4c91-8508-ad97ad1cf806-maas-api-service-ca" pod "b1224a3d-870d-4c91-8508-ad97ad1cf806" (UID: "b1224a3d-870d-4c91-8508-ad97ad1cf806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.411968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.411938 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f36e99e-f6ff-4007-a2ca-61792b5982bd-maas-api-service-ca" pod "4f36e99e-f6ff-4007-a2ca-61792b5982bd" (UID: "4f36e99e-f6ff-4007-a2ca-61792b5982bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.412437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.412409 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dfc6d0-c041-47aa-86e1-227e3b32843b-maas-api-service-ca" pod "05dfc6d0-c041-47aa-86e1-227e3b32843b" (UID: "05dfc6d0-c041-47aa-86e1-227e3b32843b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:20.415140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.415107 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f36e99e-f6ff-4007-a2ca-61792b5982bd-kube-api-access-hrg2v" pod "4f36e99e-f6ff-4007-a2ca-61792b5982bd" (UID: "4f36e99e-f6ff-4007-a2ca-61792b5982bd"). InnerVolumeSpecName "kube-api-access-hrg2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.416182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.416155 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dfc6d0-c041-47aa-86e1-227e3b32843b-kube-api-access-2rw2s" pod "05dfc6d0-c041-47aa-86e1-227e3b32843b" (UID: "05dfc6d0-c041-47aa-86e1-227e3b32843b"). InnerVolumeSpecName "kube-api-access-2rw2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.417226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.417205 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1224a3d-870d-4c91-8508-ad97ad1cf806-kube-api-access-m24vv" pod "b1224a3d-870d-4c91-8508-ad97ad1cf806" (UID: "b1224a3d-870d-4c91-8508-ad97ad1cf806"). InnerVolumeSpecName "kube-api-access-m24vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:20.514485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.514442 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrg2v\" (UniqueName: \"kubernetes.io/projected/4f36e99e-f6ff-4007-a2ca-61792b5982bd-kube-api-access-hrg2v\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.514485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.514481 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dfc6d0-c041-47aa-86e1-227e3b32843b-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.514839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.514499 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f36e99e-f6ff-4007-a2ca-61792b5982bd-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.514839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.514512 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m24vv\" (UniqueName: \"kubernetes.io/projected/b1224a3d-870d-4c91-8508-ad97ad1cf806-kube-api-access-m24vv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.514839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.514520 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1224a3d-870d-4c91-8508-ad97ad1cf806-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.514839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.514530 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rw2s\" (UniqueName: \"kubernetes.io/projected/05dfc6d0-c041-47aa-86e1-227e3b32843b-kube-api-access-2rw2s\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:20.724956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.724918 4981 generic.go:356] "Generic (PLEG): container finished" podID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerID="cc254796e14a17e09d5c097a9ef2a707314f7caf609ce7c99c794831fcc25458" exitCode=60 Nov 01 22:45:20.725425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.724988 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" event={"ID":"6c40734f-e0ed-49ef-b552-8e29e6904fa3","Type":"ContainerDied","Data":"cc254796e14a17e09d5c097a9ef2a707314f7caf609ce7c99c794831fcc25458"} Nov 01 22:45:20.725425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.725067 4981 scope.go:122] "RemoveContainer" containerID="46f9afbce6c0aebd26bb5971e4b3166c6d55090e65b93ab6172e51f8a803f276" Nov 01 22:45:20.725546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.725460 4981 scope.go:122] "RemoveContainer" containerID="cc254796e14a17e09d5c097a9ef2a707314f7caf609ce7c99c794831fcc25458" Nov 01 22:45:20.725956 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:20.725897 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd_odh-ai-gateway-infra(6c40734f-e0ed-49ef-b552-8e29e6904fa3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" Nov 01 22:45:20.727186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.727161 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" Nov 01 22:45:20.728907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.728879 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" Nov 01 22:45:20.744945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.744916 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" Nov 01 22:45:20.758648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.758180 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8459768c-6d04-42c7-a464-a6d73092db73" path="/var/lib/kubelet/pods/8459768c-6d04-42c7-a464-a6d73092db73/volumes" Nov 01 22:45:20.759527 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.759471 4981 generic.go:356] "Generic (PLEG): container finished" podID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerID="3d82eb596a29ed0237ac3221b88fe58587b021190ac387f53f5372fe16a66741" exitCode=60 Nov 01 22:45:20.760646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.760508 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" path="/var/lib/kubelet/pods/cd3eb503-df56-4cad-ad04-91b08c7f0f6b/volumes" Nov 01 22:45:20.761291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.761268 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" path="/var/lib/kubelet/pods/dc92015e-f197-4c25-ba89-fafc04e006b8/volumes" Nov 01 22:45:20.763344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.763305 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq" event={"ID":"6fc6ede6-5a8f-400a-8557-287c62afcc01","Type":"ContainerDied","Data":"cfb4614f3e0d01c4fecfb7ed7ce5b6180f5a7623e1cd003dc5fafac779987553"} Nov 01 22:45:20.763539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.763507 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn" event={"ID":"b1224a3d-870d-4c91-8508-ad97ad1cf806","Type":"ContainerDied","Data":"12ff3b4deae33cb8f1f5d493a2821ed7cc547eeb3cd531604ae982bb8544c95d"} Nov 01 22:45:20.763640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.763542 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx" event={"ID":"05dfc6d0-c041-47aa-86e1-227e3b32843b","Type":"ContainerDied","Data":"edcc02d6116ffba6c5d17c137aef6e7d5d91e65324ebf752bca549a1331eb32c"} Nov 01 22:45:20.763640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.763559 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" event={"ID":"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3","Type":"ContainerDied","Data":"3d82eb596a29ed0237ac3221b88fe58587b021190ac387f53f5372fe16a66741"} Nov 01 22:45:20.763869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.763790 4981 scope.go:122] "RemoveContainer" containerID="963ade6326d4b1ee6e0ae4186e618073f654e83e705b6cc807f422e1362c7753" Nov 01 22:45:20.764237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.764094 4981 scope.go:122] "RemoveContainer" containerID="3d82eb596a29ed0237ac3221b88fe58587b021190ac387f53f5372fe16a66741" Nov 01 22:45:20.764514 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:20.764489 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz_odh-ai-gateway-infra(2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" Nov 01 22:45:20.777142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.777115 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" event={"ID":"4f36e99e-f6ff-4007-a2ca-61792b5982bd","Type":"ContainerDied","Data":"c7f71d754da33a9d25939cb42e019b670cce92e8c5cc129f4775a6997a18908e"} Nov 01 22:45:20.777266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.777216 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4" Nov 01 22:45:20.789312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.789195 4981 scope.go:122] "RemoveContainer" containerID="977bd1558d05c2d79bcf633b427134d88bd4399b57988a3a49e597ecc139d67d" Nov 01 22:45:20.797673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.797608 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq"] Nov 01 22:45:20.807683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.807656 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-30578128-29892885-7qxmq"] Nov 01 22:45:20.807870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.807754 4981 scope.go:122] "RemoveContainer" containerID="fb1b7762463e03d185fb945e485e39b1d411276c73fbe213c6f5aec6cca43cdb" Nov 01 22:45:20.823265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.823239 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx"] Nov 01 22:45:20.832716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.832644 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-4534ad-29892885-fcmvx"] Nov 01 22:45:20.833112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.833090 4981 scope.go:122] "RemoveContainer" containerID="a040f965eb78637bf5b36bca380eccf6f808ef8e578aab699b7338569a8255ef" Nov 01 22:45:20.866653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.866155 4981 scope.go:122] "RemoveContainer" containerID="67f4ce2f1681065f1489555b49ed530febf412d061284a7464cae30f81af0518" Nov 01 22:45:20.878628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.878529 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn"] Nov 01 22:45:20.886174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.885997 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892885-npjfn"] Nov 01 22:45:20.901681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.901633 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" podUID="51de728e-ffd0-4207-9f8b-ab64505e46bc" containerName="payload-processing" containerID="cri-o://3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634" gracePeriod=25 Nov 01 22:45:20.913563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.913525 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" podUID="fa344f5f-11a6-481a-b8da-3b3a267953e6" containerName="payload-pre-processing" containerID="cri-o://32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd" gracePeriod=25 Nov 01 22:45:20.967748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:20.967716 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:21.006737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.006655 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4"] Nov 01 22:45:21.019795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.019761 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:21.042163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.033297 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892885-hsww4"] Nov 01 22:45:21.053048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.052059 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb47cc2a-e221-44bb-946c-7eca7267a452-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47cc2a-e221-44bb-946c-7eca7267a452-maas-api-service-ca\") pod \"eb47cc2a-e221-44bb-946c-7eca7267a452\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " Nov 01 22:45:21.053048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.052509 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb47cc2a-e221-44bb-946c-7eca7267a452-kube-api-access-kbs22\" (UniqueName: \"kubernetes.io/projected/eb47cc2a-e221-44bb-946c-7eca7267a452-kube-api-access-kbs22\") pod \"eb47cc2a-e221-44bb-946c-7eca7267a452\" (UID: \"eb47cc2a-e221-44bb-946c-7eca7267a452\") " Nov 01 22:45:21.056532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.056477 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb47cc2a-e221-44bb-946c-7eca7267a452-maas-api-service-ca" pod "eb47cc2a-e221-44bb-946c-7eca7267a452" (UID: "eb47cc2a-e221-44bb-946c-7eca7267a452"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:21.083813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.082105 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb47cc2a-e221-44bb-946c-7eca7267a452-kube-api-access-kbs22" pod "eb47cc2a-e221-44bb-946c-7eca7267a452" (UID: "eb47cc2a-e221-44bb-946c-7eca7267a452"). InnerVolumeSpecName "kube-api-access-kbs22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:21.155218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.155185 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb47cc2a-e221-44bb-946c-7eca7267a452-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.155218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.155218 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbs22\" (UniqueName: \"kubernetes.io/projected/eb47cc2a-e221-44bb-946c-7eca7267a452-kube-api-access-kbs22\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.183182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.183148 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:21.187064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.187041 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:21.256558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.256528 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce1efe76-4df3-492e-be84-4c58ad778677-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1efe76-4df3-492e-be84-4c58ad778677-maas-api-service-ca\") pod \"ce1efe76-4df3-492e-be84-4c58ad778677\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " Nov 01 22:45:21.256824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.256639 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8472535c-7063-4bea-82b9-39a6ee919bc0-kube-api-access-f25rp\" (UniqueName: \"kubernetes.io/projected/8472535c-7063-4bea-82b9-39a6ee919bc0-kube-api-access-f25rp\") pod \"8472535c-7063-4bea-82b9-39a6ee919bc0\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " Nov 01 22:45:21.256824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.256776 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce1efe76-4df3-492e-be84-4c58ad778677-kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/ce1efe76-4df3-492e-be84-4c58ad778677-kube-api-access-ndxbp\") pod \"ce1efe76-4df3-492e-be84-4c58ad778677\" (UID: \"ce1efe76-4df3-492e-be84-4c58ad778677\") " Nov 01 22:45:21.256952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.256884 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8472535c-7063-4bea-82b9-39a6ee919bc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8472535c-7063-4bea-82b9-39a6ee919bc0-maas-api-service-ca\") pod \"8472535c-7063-4bea-82b9-39a6ee919bc0\" (UID: \"8472535c-7063-4bea-82b9-39a6ee919bc0\") " Nov 01 22:45:21.257007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.256948 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce1efe76-4df3-492e-be84-4c58ad778677-maas-api-service-ca" pod "ce1efe76-4df3-492e-be84-4c58ad778677" (UID: "ce1efe76-4df3-492e-be84-4c58ad778677"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:21.257415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.257392 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8472535c-7063-4bea-82b9-39a6ee919bc0-maas-api-service-ca" pod "8472535c-7063-4bea-82b9-39a6ee919bc0" (UID: "8472535c-7063-4bea-82b9-39a6ee919bc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:21.257771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.257755 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8472535c-7063-4bea-82b9-39a6ee919bc0-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.257829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.257773 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce1efe76-4df3-492e-be84-4c58ad778677-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.259405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.259381 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1efe76-4df3-492e-be84-4c58ad778677-kube-api-access-ndxbp" pod "ce1efe76-4df3-492e-be84-4c58ad778677" (UID: "ce1efe76-4df3-492e-be84-4c58ad778677"). InnerVolumeSpecName "kube-api-access-ndxbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:21.259516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.259456 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8472535c-7063-4bea-82b9-39a6ee919bc0-kube-api-access-f25rp" pod "8472535c-7063-4bea-82b9-39a6ee919bc0" (UID: "8472535c-7063-4bea-82b9-39a6ee919bc0"). InnerVolumeSpecName "kube-api-access-f25rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:21.359173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.359143 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f25rp\" (UniqueName: \"kubernetes.io/projected/8472535c-7063-4bea-82b9-39a6ee919bc0-kube-api-access-f25rp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.359173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.359170 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndxbp\" (UniqueName: \"kubernetes.io/projected/ce1efe76-4df3-492e-be84-4c58ad778677-kube-api-access-ndxbp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:21.689019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.688979 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" podUID="51de728e-ffd0-4207-9f8b-ab64505e46bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:21.716959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.716929 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" podUID="fa344f5f-11a6-481a-b8da-3b3a267953e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:21.777985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.777952 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd"] Nov 01 22:45:21.795770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.795745 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" Nov 01 22:45:21.795770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.795766 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4" event={"ID":"eb47cc2a-e221-44bb-946c-7eca7267a452","Type":"ContainerDied","Data":"9aab46f1d38eade117d57e561acfab1b99a1d2f290efbde9f27db417349e35b7"} Nov 01 22:45:21.796058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.795824 4981 scope.go:122] "RemoveContainer" containerID="c9f4cfa652b6e2763cc647f73e3687d0e55c072a6cf60360ad155a152c83c9f2" Nov 01 22:45:21.804008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.803987 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" Nov 01 22:45:21.804120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.804036 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp" event={"ID":"ce1efe76-4df3-492e-be84-4c58ad778677","Type":"ContainerDied","Data":"35b62196cce61f48bdca9fd866314e5f88dfa3790038c19c368925a2b3300d2a"} Nov 01 22:45:21.805533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.805455 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" event={"ID":"8472535c-7063-4bea-82b9-39a6ee919bc0","Type":"ContainerDied","Data":"efda556e8e22f6233ff6680b92e60384366af2c5d032787040d6209b2086d9bf"} Nov 01 22:45:21.805533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.805518 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg" Nov 01 22:45:21.811648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.811426 4981 scope.go:122] "RemoveContainer" containerID="a642c55a97b39942e173d08d4e788e14453ad10d023be9031c8fc66ef90ca271" Nov 01 22:45:21.825950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.825927 4981 scope.go:122] "RemoveContainer" containerID="b82147bc1be13ea1648722b2d9fa40d862af148134c33dbc391b6c61049c882a" Nov 01 22:45:21.832253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.832229 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4"] Nov 01 22:45:21.837950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.837929 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892885-6czt4"] Nov 01 22:45:21.884937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.884714 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg"] Nov 01 22:45:21.896937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.896910 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-927b68-29892885-7wfhg"] Nov 01 22:45:21.917139 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.917098 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp"] Nov 01 22:45:21.924497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.924465 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892885-6lnbp"] Nov 01 22:45:21.985882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.985837 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz"] Nov 01 22:45:21.992969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.992941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:21.995668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:21.995644 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:22.072983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.072947 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c40734f-e0ed-49ef-b552-8e29e6904fa3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40734f-e0ed-49ef-b552-8e29e6904fa3-maas-api-service-ca\") pod \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " Nov 01 22:45:22.073212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.073196 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c40734f-e0ed-49ef-b552-8e29e6904fa3-kube-api-access-vqnqx\" (UniqueName: \"kubernetes.io/projected/6c40734f-e0ed-49ef-b552-8e29e6904fa3-kube-api-access-vqnqx\") pod \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\" (UID: \"6c40734f-e0ed-49ef-b552-8e29e6904fa3\") " Nov 01 22:45:22.075044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.075016 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c40734f-e0ed-49ef-b552-8e29e6904fa3-maas-api-service-ca" pod "6c40734f-e0ed-49ef-b552-8e29e6904fa3" (UID: "6c40734f-e0ed-49ef-b552-8e29e6904fa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:22.075486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.075462 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40734f-e0ed-49ef-b552-8e29e6904fa3-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.077372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.077301 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c40734f-e0ed-49ef-b552-8e29e6904fa3-kube-api-access-vqnqx" pod "6c40734f-e0ed-49ef-b552-8e29e6904fa3" (UID: "6c40734f-e0ed-49ef-b552-8e29e6904fa3"). InnerVolumeSpecName "kube-api-access-vqnqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:22.177523 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.177490 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqnqx\" (UniqueName: \"kubernetes.io/projected/6c40734f-e0ed-49ef-b552-8e29e6904fa3-kube-api-access-vqnqx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.236855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.236796 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:22.279788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.279761 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-kube-api-access-zszft\" (UniqueName: \"kubernetes.io/projected/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-kube-api-access-zszft\") pod \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " Nov 01 22:45:22.279991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.279971 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-maas-api-service-ca\") pod \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\" (UID: \"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3\") " Nov 01 22:45:22.280399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.280358 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-maas-api-service-ca" pod "2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" (UID: "2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:22.280706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.280688 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.282048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.282020 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-kube-api-access-zszft" pod "2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" (UID: "2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3"). InnerVolumeSpecName "kube-api-access-zszft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:22.382695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.382661 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszft\" (UniqueName: \"kubernetes.io/projected/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3-kube-api-access-zszft\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:22.689026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.688998 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:22.756302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.756263 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" path="/var/lib/kubelet/pods/05dfc6d0-c041-47aa-86e1-227e3b32843b/volumes" Nov 01 22:45:22.757338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.757307 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" path="/var/lib/kubelet/pods/4f36e99e-f6ff-4007-a2ca-61792b5982bd/volumes" Nov 01 22:45:22.758009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.757984 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" path="/var/lib/kubelet/pods/6fc6ede6-5a8f-400a-8557-287c62afcc01/volumes" Nov 01 22:45:22.759309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.759263 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" path="/var/lib/kubelet/pods/8472535c-7063-4bea-82b9-39a6ee919bc0/volumes" Nov 01 22:45:22.760260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.760231 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" path="/var/lib/kubelet/pods/b1224a3d-870d-4c91-8508-ad97ad1cf806/volumes" Nov 01 22:45:22.760942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.760918 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" path="/var/lib/kubelet/pods/ce1efe76-4df3-492e-be84-4c58ad778677/volumes" Nov 01 22:45:22.762432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.762406 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" path="/var/lib/kubelet/pods/eb47cc2a-e221-44bb-946c-7eca7267a452/volumes" Nov 01 22:45:22.828121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.828092 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" Nov 01 22:45:22.828121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.828113 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd" event={"ID":"6c40734f-e0ed-49ef-b552-8e29e6904fa3","Type":"ContainerDied","Data":"af2a24fc9cd57db2c8d4f181ce22cf0eb7d09fe7e77caffd9a54cb3e72d65fb8"} Nov 01 22:45:22.830044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.828176 4981 scope.go:122] "RemoveContainer" containerID="cc254796e14a17e09d5c097a9ef2a707314f7caf609ce7c99c794831fcc25458" Nov 01 22:45:22.833133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.833081 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" Nov 01 22:45:22.833265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.833175 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz" event={"ID":"2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3","Type":"ContainerDied","Data":"5fe0a3ee231326a0fe93df025a665188c539bb27951f7687b7464945e8a54a7d"} Nov 01 22:45:22.846877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.846853 4981 scope.go:122] "RemoveContainer" containerID="3d82eb596a29ed0237ac3221b88fe58587b021190ac387f53f5372fe16a66741" Nov 01 22:45:22.860668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.860635 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz"] Nov 01 22:45:22.867845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.867720 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892885-527lz"] Nov 01 22:45:22.892074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.892041 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd"] Nov 01 22:45:22.911793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:22.911761 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ee07ad-29892885-mkvhd"] Nov 01 22:45:23.635487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:23.635450 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:24.049283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.049251 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm"] Nov 01 22:45:24.054789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.054760 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx"] Nov 01 22:45:24.060206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.060181 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r"] Nov 01 22:45:24.064858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.064832 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf9828c7-jkgwm"] Nov 01 22:45:24.069461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.069441 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9269d490-8tgjx"] Nov 01 22:45:24.073548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.073528 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a51171-25g6r"] Nov 01 22:45:24.440966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.440938 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:24.747159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.747073 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" path="/var/lib/kubelet/pods/2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3/volumes" Nov 01 22:45:24.748123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.748070 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337a224f-077a-4646-b0a3-ce2a7c1179a8" path="/var/lib/kubelet/pods/337a224f-077a-4646-b0a3-ce2a7c1179a8/volumes" Nov 01 22:45:24.748907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.748882 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" path="/var/lib/kubelet/pods/6c40734f-e0ed-49ef-b552-8e29e6904fa3/volumes" Nov 01 22:45:24.749982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.749963 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db5039a-aa92-4889-b66e-187cbdaefdb7" path="/var/lib/kubelet/pods/6db5039a-aa92-4889-b66e-187cbdaefdb7/volumes" Nov 01 22:45:24.750421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:24.750406 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c451365c-af96-46e5-8413-8e73372f7446" path="/var/lib/kubelet/pods/c451365c-af96-46e5-8413-8e73372f7446/volumes" Nov 01 22:45:25.425885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:25.425853 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:26.071457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:26.071424 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:27.070597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.053949 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:27.090660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.090626 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk"] Nov 01 22:45:27.158958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.138317 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24888703-bmnkk"] Nov 01 22:45:27.167260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.164412 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:27.379651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.379012 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:27.510601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.505235 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:27.996687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.996657 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv"] Nov 01 22:45:27.997840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997822 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.997840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997840 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997849 4981 cpu_manager.go:414] "containerMap: removing container" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997856 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997868 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997876 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997889 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997897 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997916 4981 cpu_manager.go:414] "containerMap: removing container" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997923 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:27.997934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997934 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997941 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997968 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997976 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997991 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.997996 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998010 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998015 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998021 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998026 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998039 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" containerName="revoke-keys" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998047 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" containerName="revoke-keys" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998059 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998064 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998069 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998073 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998081 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998086 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998095 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998099 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998107 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998111 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998118 4981 cpu_manager.go:414] "containerMap: removing container" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998123 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998133 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998138 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998148 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998153 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998158 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998163 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998172 4981 cpu_manager.go:414] "containerMap: removing container" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998177 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998187 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:27.998183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998192 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998201 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998205 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998213 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998217 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998223 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998227 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998236 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998240 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998368 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998376 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998384 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998390 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998395 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998401 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" containerName="revoke-keys" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998407 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998412 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998417 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998422 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998428 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998433 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998438 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998445 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998452 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998457 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998462 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998468 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998473 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998478 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998482 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998488 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998493 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998498 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998503 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8472535c-7063-4bea-82b9-39a6ee919bc0" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998508 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998514 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c40734f-e0ed-49ef-b552-8e29e6904fa3" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998521 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:27.999090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:27.998526 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:28.004310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.004282 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.008248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.008226 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv"] Nov 01 22:45:28.010027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.010004 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:45:28.010140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.010053 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:45:28.031656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.031624 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:45:28.036294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.036261 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:45:28.142642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.142606 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcz4w\" (UniqueName: \"kubernetes.io/projected/b9836a54-5722-455f-9e92-2a085f4a2c3b-kube-api-access-kcz4w\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.143014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.142849 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.143054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.143025 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.245144 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.245102 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.245341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.245237 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.245446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.245426 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcz4w\" (UniqueName: \"kubernetes.io/projected/b9836a54-5722-455f-9e92-2a085f4a2c3b-kube-api-access-kcz4w\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.245995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.245967 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.247845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.247795 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.255802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.255781 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcz4w\" (UniqueName: \"kubernetes.io/projected/b9836a54-5722-455f-9e92-2a085f4a2c3b-kube-api-access-kcz4w\") pod \"maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.346406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.346375 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:28.562566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.562492 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:28.646648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.646621 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv"] Nov 01 22:45:28.647106 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:28.647082 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9836a54_5722_455f_9e92_2a085f4a2c3b.slice/crio-2117301c5240ed5c665ae9ece8af5547421e657717754584579ee6e956f2dd23 WatchSource:0}: Error finding container 2117301c5240ed5c665ae9ece8af5547421e657717754584579ee6e956f2dd23: Status 404 returned error can't find the container with id 2117301c5240ed5c665ae9ece8af5547421e657717754584579ee6e956f2dd23 Nov 01 22:45:28.738506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.738473 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc0d0707-bf21-41c9-9eff-ea900c0b8a9c" path="/var/lib/kubelet/pods/bc0d0707-bf21-41c9-9eff-ea900c0b8a9c/volumes" Nov 01 22:45:28.902751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:28.902721 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" event={"ID":"b9836a54-5722-455f-9e92-2a085f4a2c3b","Type":"ContainerStarted","Data":"2117301c5240ed5c665ae9ece8af5547421e657717754584579ee6e956f2dd23"} Nov 01 22:45:29.520636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:29.520609 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:29.913392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:29.913305 4981 generic.go:356] "Generic (PLEG): container finished" podID="b9836a54-5722-455f-9e92-2a085f4a2c3b" containerID="666b723a8b260ea71542f399f395fd25688813e720ea8525a5d45e8aa07f3310" exitCode=0 Nov 01 22:45:29.913550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:29.913394 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" event={"ID":"b9836a54-5722-455f-9e92-2a085f4a2c3b","Type":"ContainerDied","Data":"666b723a8b260ea71542f399f395fd25688813e720ea8525a5d45e8aa07f3310"} Nov 01 22:45:30.203025 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.202993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:30.886016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.885986 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz"] Nov 01 22:45:30.888047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888015 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:30.888047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888047 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc92015e-f197-4c25-ba89-fafc04e006b8" containerName="cleanup" Nov 01 22:45:30.888260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888082 4981 cpu_manager.go:414] "containerMap: removing container" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:30.888260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888096 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:30.888260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888185 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:30.888260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888193 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:30.888260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888233 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:30.888260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888241 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce1efe76-4df3-492e-be84-4c58ad778677" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888263 4981 cpu_manager.go:414] "containerMap: removing container" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888271 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888299 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888307 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc6ede6-5a8f-400a-8557-287c62afcc01" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888336 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888344 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3eb503-df56-4cad-ad04-91b08c7f0f6b" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888386 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:30.888552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888394 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:30.888983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888862 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dfc6d0-c041-47aa-86e1-227e3b32843b" containerName="cleanup" Nov 01 22:45:30.888983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888905 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb47cc2a-e221-44bb-946c-7eca7267a452" containerName="cleanup" Nov 01 22:45:30.888983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888930 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f36e99e-f6ff-4007-a2ca-61792b5982bd" containerName="cleanup" Nov 01 22:45:30.888983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888952 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1224a3d-870d-4c91-8508-ad97ad1cf806" containerName="cleanup" Nov 01 22:45:30.888983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.888968 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3a5cbc-b1bd-4ca3-9f69-af0bff0cefd3" containerName="cleanup" Nov 01 22:45:30.894923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.894898 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:30.908854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.908821 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz"] Nov 01 22:45:30.933907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:30.933831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:31.014482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.014437 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.014697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.014666 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4468\" (UniqueName: \"kubernetes.io/projected/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-kube-api-access-k4468\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.014830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.014813 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.117411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.117365 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4468\" (UniqueName: \"kubernetes.io/projected/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-kube-api-access-k4468\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.117671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.117551 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.117875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.117858 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.118872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.118840 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.120538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.120515 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.132389 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.132362 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4468\" (UniqueName: \"kubernetes.io/projected/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-kube-api-access-k4468\") pod \"maas-api-revoke-sub-534f6d86-lfmrz\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.166356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.166329 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:31.190000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.189963 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:31.216816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.216784 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:31.321426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.321395 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-openshift-ca\") pod \"b9836a54-5722-455f-9e92-2a085f4a2c3b\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " Nov 01 22:45:31.321644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.321510 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9836a54-5722-455f-9e92-2a085f4a2c3b-kube-api-access-kcz4w\" (UniqueName: \"kubernetes.io/projected/b9836a54-5722-455f-9e92-2a085f4a2c3b-kube-api-access-kcz4w\") pod \"b9836a54-5722-455f-9e92-2a085f4a2c3b\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " Nov 01 22:45:31.321716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.321656 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-xks-ca\") pod \"b9836a54-5722-455f-9e92-2a085f4a2c3b\" (UID: \"b9836a54-5722-455f-9e92-2a085f4a2c3b\") " Nov 01 22:45:31.321867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.321782 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-openshift-ca" pod "b9836a54-5722-455f-9e92-2a085f4a2c3b" (UID: "b9836a54-5722-455f-9e92-2a085f4a2c3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:31.322419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.322389 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:31.324284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.324260 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9836a54-5722-455f-9e92-2a085f4a2c3b-kube-api-access-kcz4w" pod "b9836a54-5722-455f-9e92-2a085f4a2c3b" (UID: "b9836a54-5722-455f-9e92-2a085f4a2c3b"). InnerVolumeSpecName "kube-api-access-kcz4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:31.325588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.325556 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-xks-ca" pod "b9836a54-5722-455f-9e92-2a085f4a2c3b" (UID: "b9836a54-5722-455f-9e92-2a085f4a2c3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:31.425688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.424225 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9836a54-5722-455f-9e92-2a085f4a2c3b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:31.425688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.424258 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcz4w\" (UniqueName: \"kubernetes.io/projected/b9836a54-5722-455f-9e92-2a085f4a2c3b-kube-api-access-kcz4w\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:31.598096 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:31.598062 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df6aa13_4de6_4fc3_b3dd_fc96a09bb77f.slice/crio-1de6d546508566f27e279710bc438f7e33deb92aeb2b48e8771fe65e6f31938c WatchSource:0}: Error finding container 1de6d546508566f27e279710bc438f7e33deb92aeb2b48e8771fe65e6f31938c: Status 404 returned error can't find the container with id 1de6d546508566f27e279710bc438f7e33deb92aeb2b48e8771fe65e6f31938c Nov 01 22:45:31.598553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.598528 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz"] Nov 01 22:45:31.688969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.688878 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" podUID="51de728e-ffd0-4207-9f8b-ab64505e46bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:31.717121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.717088 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" podUID="fa344f5f-11a6-481a-b8da-3b3a267953e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:45:31.940876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.940788 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" Nov 01 22:45:31.940876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.940816 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv" event={"ID":"b9836a54-5722-455f-9e92-2a085f4a2c3b","Type":"ContainerDied","Data":"2117301c5240ed5c665ae9ece8af5547421e657717754584579ee6e956f2dd23"} Nov 01 22:45:31.940876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.940864 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2117301c5240ed5c665ae9ece8af5547421e657717754584579ee6e956f2dd23" Nov 01 22:45:31.942487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:31.942462 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" event={"ID":"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f","Type":"ContainerStarted","Data":"1de6d546508566f27e279710bc438f7e33deb92aeb2b48e8771fe65e6f31938c"} Nov 01 22:45:32.104166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.104141 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:32.318753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.318675 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:32.581373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.581297 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56"] Nov 01 22:45:32.582629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.582553 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b9836a54-5722-455f-9e92-2a085f4a2c3b" containerName="revoke-keys" Nov 01 22:45:32.582629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.582594 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9836a54-5722-455f-9e92-2a085f4a2c3b" containerName="revoke-keys" Nov 01 22:45:32.583141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.582844 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9836a54-5722-455f-9e92-2a085f4a2c3b" containerName="revoke-keys" Nov 01 22:45:32.587371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.587350 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.599466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.599442 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2476f8-openshift-default-dockercfg-jrkp5\"" Nov 01 22:45:32.603645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.603619 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56"] Nov 01 22:45:32.609104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.609075 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645317 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-envoy\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645403 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-certs\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645533 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45x5\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-kube-api-access-t45x5\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645661 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-credential-socket\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645732 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-socket\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645768 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-token\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645816 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-podinfo\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34390f-1892-407b-b5a6-bc3184b7bc09-istiod-ca-cert\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.652230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.645974 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-data\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.767951 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45x5\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-kube-api-access-t45x5\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768059 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-credential-socket\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768116 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-socket\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768150 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-token\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768191 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-podinfo\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768270 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34390f-1892-407b-b5a6-bc3184b7bc09-istiod-ca-cert\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768322 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-data\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768469 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-envoy\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.803974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.768526 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-certs\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.821224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.811854 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-credential-socket\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.830795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.830753 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-socket\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.831768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.831701 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34390f-1892-407b-b5a6-bc3184b7bc09-istiod-ca-cert\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.856147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.856108 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-data\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.856759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.856714 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-certs\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.923428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.907210 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:32.951148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.951116 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-envoy\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.977600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.975472 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-podinfo\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.987361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.987286 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45x5\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-kube-api-access-t45x5\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:32.988040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:32.987969 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-token\") pod \"e2e-api-a-2476f8-openshift-default-8489fc489f-79b56\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:33.035983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.035920 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" podStartSLOduration=3.03590337 podStartE2EDuration="3.03590337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:33.033980799 +0000 UTC m=+1298.461858733" watchObservedRunningTime="2026-11-01 22:45:33.03590337 +0000 UTC m=+1298.463781308" Nov 01 22:45:33.056426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.056383 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" event={"ID":"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f","Type":"ContainerStarted","Data":"89c6221cd614baf63b15ef8aaca1f8162ad8e0b6eb93383462a9a317fb180f1e"} Nov 01 22:45:33.208445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.208407 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:33.537368 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:33.537333 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a34390f_1892_407b_b5a6_bc3184b7bc09.slice/crio-abf08e6272726bed030878cd5db7f4cdb7c4228906dc19f94f3a27b7cfa16d96 WatchSource:0}: Error finding container abf08e6272726bed030878cd5db7f4cdb7c4228906dc19f94f3a27b7cfa16d96: Status 404 returned error can't find the container with id abf08e6272726bed030878cd5db7f4cdb7c4228906dc19f94f3a27b7cfa16d96 Nov 01 22:45:33.539843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.539819 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56"] Nov 01 22:45:33.540880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.540848 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:33.540973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.540928 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:33.541046 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.540970 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:33.595598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:33.595452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:34.009367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.009327 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" event={"ID":"5a34390f-1892-407b-b5a6-bc3184b7bc09","Type":"ContainerStarted","Data":"e8683695c4b043a8ac74442d9fcc7499a54b39eacfbcd95b65c867a48b674f20"} Nov 01 22:45:34.009891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.009390 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" event={"ID":"5a34390f-1892-407b-b5a6-bc3184b7bc09","Type":"ContainerStarted","Data":"abf08e6272726bed030878cd5db7f4cdb7c4228906dc19f94f3a27b7cfa16d96"} Nov 01 22:45:34.011295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.011272 4981 generic.go:356] "Generic (PLEG): container finished" podID="3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" containerID="89c6221cd614baf63b15ef8aaca1f8162ad8e0b6eb93383462a9a317fb180f1e" exitCode=0 Nov 01 22:45:34.011497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.011375 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" event={"ID":"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f","Type":"ContainerDied","Data":"89c6221cd614baf63b15ef8aaca1f8162ad8e0b6eb93383462a9a317fb180f1e"} Nov 01 22:45:34.034336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.034284 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podStartSLOduration=2.034270616 podStartE2EDuration="2.034270616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:34.033379906 +0000 UTC m=+1299.461257877" watchObservedRunningTime="2026-11-01 22:45:34.034270616 +0000 UTC m=+1299.462148546" Nov 01 22:45:34.211813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.209200 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:34.218148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.218116 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:34.396613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:34.396518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:35.023269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.023232 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:35.024635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.024273 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:45:35.125278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.125245 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:35.399506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.399479 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:35.436235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.436200 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-xks-ca\") pod \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " Nov 01 22:45:35.436460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.436437 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-kube-api-access-k4468\" (UniqueName: \"kubernetes.io/projected/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-kube-api-access-k4468\") pod \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " Nov 01 22:45:35.436657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.436604 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-openshift-ca\") pod \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\" (UID: \"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f\") " Nov 01 22:45:35.437828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.437794 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-openshift-ca" pod "3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" (UID: "3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:35.440528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.440497 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-kube-api-access-k4468" pod "3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" (UID: "3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f"). InnerVolumeSpecName "kube-api-access-k4468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:35.458736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.458706 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-xks-ca" pod "3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" (UID: "3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:45:35.539512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.539480 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:35.539512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.539509 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4468\" (UniqueName: \"kubernetes.io/projected/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-kube-api-access-k4468\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:35.539727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:35.539520 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:36.036400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:36.036367 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" Nov 01 22:45:36.036400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:36.036384 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz" event={"ID":"3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f","Type":"ContainerDied","Data":"1de6d546508566f27e279710bc438f7e33deb92aeb2b48e8771fe65e6f31938c"} Nov 01 22:45:36.036948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:36.036446 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de6d546508566f27e279710bc438f7e33deb92aeb2b48e8771fe65e6f31938c" Nov 01 22:45:36.094722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:36.094686 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:36.344986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:36.344905 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:36.481300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:36.481266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:37.036647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:37.036609 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr"] Nov 01 22:45:37.045324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:37.045294 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch"] Nov 01 22:45:37.063250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:37.063211 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c024f-l7xjr"] Nov 01 22:45:37.068681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:37.068635 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9734fb-hb9ch"] Nov 01 22:45:37.489898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:37.489816 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:37.655054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:37.655013 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:37.806491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:37.806398 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:38.540107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:38.540072 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl"] Nov 01 22:45:38.675934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:38.675902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:38.743257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:38.743222 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b054b7-b214-4468-9787-09dad5ff2abe" path="/var/lib/kubelet/pods/83b054b7-b214-4468-9787-09dad5ff2abe/volumes" Nov 01 22:45:38.743911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:38.743888 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5" path="/var/lib/kubelet/pods/99d32a98-a3ef-4ae7-ac4e-c49d9fe9dac5/volumes" Nov 01 22:45:38.799176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:38.799071 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:38.928006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:38.927969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:39.911288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:39.911254 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:40.713808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:40.713771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:41.275987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.275963 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:45:41.280069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.280049 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:45:41.304025 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.303997 4981 generic.go:356] "Generic (PLEG): container finished" podID="51de728e-ffd0-4207-9f8b-ab64505e46bc" containerID="3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634" exitCode=0 Nov 01 22:45:41.304711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.304035 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" Nov 01 22:45:41.304711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.304081 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" event={"ID":"51de728e-ffd0-4207-9f8b-ab64505e46bc","Type":"ContainerDied","Data":"3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634"} Nov 01 22:45:41.304711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.304130 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5" event={"ID":"51de728e-ffd0-4207-9f8b-ab64505e46bc","Type":"ContainerDied","Data":"a35e8e7a80143c77a0450b95a7b9bf43985c006bc4144b84643d6e4ce02ba53d"} Nov 01 22:45:41.304711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.304161 4981 scope.go:122] "RemoveContainer" containerID="3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634" Nov 01 22:45:41.307507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.306836 4981 generic.go:356] "Generic (PLEG): container finished" podID="fa344f5f-11a6-481a-b8da-3b3a267953e6" containerID="32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd" exitCode=0 Nov 01 22:45:41.307507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.307192 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" event={"ID":"fa344f5f-11a6-481a-b8da-3b3a267953e6","Type":"ContainerDied","Data":"32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd"} Nov 01 22:45:41.307507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.307240 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" event={"ID":"fa344f5f-11a6-481a-b8da-3b3a267953e6","Type":"ContainerDied","Data":"dddd24760bc54d4b55ff8fa6e473fc06cc1c2da5b0f81d1d9f0371af78729e79"} Nov 01 22:45:41.307507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.307313 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z" Nov 01 22:45:41.320523 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.320499 4981 scope.go:122] "RemoveContainer" containerID="3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634" Nov 01 22:45:41.321372 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:41.320987 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634\": container with ID starting with 3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634 not found: ID does not exist" containerID="3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634" Nov 01 22:45:41.321372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.321032 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634"} err="failed to get container status \"3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634\": rpc error: code = NotFound desc = could not find container \"3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634\": container with ID starting with 3fac07628082e0d593efb369aadc98a3637163ad3132c3d8262c77804ccf2634 not found: ID does not exist" Nov 01 22:45:41.321372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.321056 4981 scope.go:122] "RemoveContainer" containerID="32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd" Nov 01 22:45:41.330213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.330173 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa344f5f-11a6-481a-b8da-3b3a267953e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa344f5f-11a6-481a-b8da-3b3a267953e6-plugins-config-volume\") pod \"fa344f5f-11a6-481a-b8da-3b3a267953e6\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " Nov 01 22:45:41.335135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.330448 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51de728e-ffd0-4207-9f8b-ab64505e46bc-kube-api-access-r6l4l\" (UniqueName: \"kubernetes.io/projected/51de728e-ffd0-4207-9f8b-ab64505e46bc-kube-api-access-r6l4l\") pod \"51de728e-ffd0-4207-9f8b-ab64505e46bc\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " Nov 01 22:45:41.335135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.330541 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51de728e-ffd0-4207-9f8b-ab64505e46bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51de728e-ffd0-4207-9f8b-ab64505e46bc-plugins-config-volume\") pod \"51de728e-ffd0-4207-9f8b-ab64505e46bc\" (UID: \"51de728e-ffd0-4207-9f8b-ab64505e46bc\") " Nov 01 22:45:41.335135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.330630 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa344f5f-11a6-481a-b8da-3b3a267953e6-kube-api-access-5hfsg\" (UniqueName: \"kubernetes.io/projected/fa344f5f-11a6-481a-b8da-3b3a267953e6-kube-api-access-5hfsg\") pod \"fa344f5f-11a6-481a-b8da-3b3a267953e6\" (UID: \"fa344f5f-11a6-481a-b8da-3b3a267953e6\") " Nov 01 22:45:41.335135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.332280 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51de728e-ffd0-4207-9f8b-ab64505e46bc-plugins-config-volume" pod "51de728e-ffd0-4207-9f8b-ab64505e46bc" (UID: "51de728e-ffd0-4207-9f8b-ab64505e46bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:41.335135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.334885 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa344f5f-11a6-481a-b8da-3b3a267953e6-plugins-config-volume" pod "fa344f5f-11a6-481a-b8da-3b3a267953e6" (UID: "fa344f5f-11a6-481a-b8da-3b3a267953e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:41.344727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.343554 4981 scope.go:122] "RemoveContainer" containerID="32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd" Nov 01 22:45:41.344727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.343657 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa344f5f-11a6-481a-b8da-3b3a267953e6-kube-api-access-5hfsg" pod "fa344f5f-11a6-481a-b8da-3b3a267953e6" (UID: "fa344f5f-11a6-481a-b8da-3b3a267953e6"). InnerVolumeSpecName "kube-api-access-5hfsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:41.344727 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:41.343988 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd\": container with ID starting with 32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd not found: ID does not exist" containerID="32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd" Nov 01 22:45:41.344727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.344018 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd"} err="failed to get container status \"32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd\": rpc error: code = NotFound desc = could not find container \"32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd\": container with ID starting with 32cbd49155e99fd69087352a09ead41191f631fb308301ca1e3791729cd1acdd not found: ID does not exist" Nov 01 22:45:41.344727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.344565 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51de728e-ffd0-4207-9f8b-ab64505e46bc-kube-api-access-r6l4l" pod "51de728e-ffd0-4207-9f8b-ab64505e46bc" (UID: "51de728e-ffd0-4207-9f8b-ab64505e46bc"). InnerVolumeSpecName "kube-api-access-r6l4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:41.423238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.422173 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:41.441282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.439334 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6l4l\" (UniqueName: \"kubernetes.io/projected/51de728e-ffd0-4207-9f8b-ab64505e46bc-kube-api-access-r6l4l\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:41.441282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.439370 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51de728e-ffd0-4207-9f8b-ab64505e46bc-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:41.441282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.439388 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hfsg\" (UniqueName: \"kubernetes.io/projected/fa344f5f-11a6-481a-b8da-3b3a267953e6-kube-api-access-5hfsg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:41.441282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.439402 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa344f5f-11a6-481a-b8da-3b3a267953e6-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:41.633232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.633203 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5"] Nov 01 22:45:41.637969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.637943 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-l7qh5"] Nov 01 22:45:41.652177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.652151 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z"] Nov 01 22:45:41.664063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:41.664041 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-30578128-796c56f57f-qkm8z"] Nov 01 22:45:42.290357 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.290271 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:42.563795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.563707 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:42.738935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.738898 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51de728e-ffd0-4207-9f8b-ab64505e46bc" path="/var/lib/kubelet/pods/51de728e-ffd0-4207-9f8b-ab64505e46bc/volumes" Nov 01 22:45:42.739409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.739391 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa344f5f-11a6-481a-b8da-3b3a267953e6" path="/var/lib/kubelet/pods/fa344f5f-11a6-481a-b8da-3b3a267953e6/volumes" Nov 01 22:45:42.972957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.972929 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z"] Nov 01 22:45:42.973743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973727 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" containerName="revoke-keys" Nov 01 22:45:42.973788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973744 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" containerName="revoke-keys" Nov 01 22:45:42.973788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973774 4981 cpu_manager.go:414] "containerMap: removing container" podUID="51de728e-ffd0-4207-9f8b-ab64505e46bc" containerName="payload-processing" Nov 01 22:45:42.973788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973780 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51de728e-ffd0-4207-9f8b-ab64505e46bc" containerName="payload-processing" Nov 01 22:45:42.973904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973815 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fa344f5f-11a6-481a-b8da-3b3a267953e6" containerName="payload-pre-processing" Nov 01 22:45:42.973904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973820 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa344f5f-11a6-481a-b8da-3b3a267953e6" containerName="payload-pre-processing" Nov 01 22:45:42.973967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973937 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="51de728e-ffd0-4207-9f8b-ab64505e46bc" containerName="payload-processing" Nov 01 22:45:42.973967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973950 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa344f5f-11a6-481a-b8da-3b3a267953e6" containerName="payload-pre-processing" Nov 01 22:45:42.973967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.973961 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" containerName="revoke-keys" Nov 01 22:45:42.977945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.977927 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:42.984206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:42.984185 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c6cec6-openshift-default-dockercfg-267mk\"" Nov 01 22:45:43.003061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.003033 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z"] Nov 01 22:45:43.095002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.088359 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-socket\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.133558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.112103 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:43.304773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.304683 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-token\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.305216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.304836 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c078984-19f8-4130-9ec5-6ae240e8557d-istiod-ca-cert\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.305216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.304910 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-envoy\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.305216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.305028 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-credential-socket\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.305216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.305118 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-certs\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.305216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.305207 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-data\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.305671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.305312 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-podinfo\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.305671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.305394 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dvk\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-kube-api-access-s6dvk\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-data\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407227 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-podinfo\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407277 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dvk\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-kube-api-access-s6dvk\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407323 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-socket\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407411 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-token\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407482 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c078984-19f8-4130-9ec5-6ae240e8557d-istiod-ca-cert\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407517 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-envoy\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407606 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-credential-socket\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.407844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.407667 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-certs\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.408094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.408074 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-certs\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.408308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.408289 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-data\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.408987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.408960 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-credential-socket\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.409850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.409824 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c078984-19f8-4130-9ec5-6ae240e8557d-istiod-ca-cert\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.409968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.409893 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-socket\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.411078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.411053 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:43.411832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.411807 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-envoy\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.412307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.412280 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-podinfo\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.422550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.422515 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-token\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.423627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.423562 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dvk\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-kube-api-access-s6dvk\") pod \"e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.541034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.540977 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" podUID="f3f42c4d-cebd-462d-a59c-4252fa0b8e78" containerName="payload-processing" containerID="cri-o://21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33" gracePeriod=25 Nov 01 22:45:43.594547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.594521 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:43.772939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.772910 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:43.917632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.917553 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z"] Nov 01 22:45:43.918409 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:43.918382 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c078984_19f8_4130_9ec5_6ae240e8557d.slice/crio-f21eb61636b5ec0b708760296902e7a6be878b8ed4dcab0928a83b07d3784826 WatchSource:0}: Error finding container f21eb61636b5ec0b708760296902e7a6be878b8ed4dcab0928a83b07d3784826: Status 404 returned error can't find the container with id f21eb61636b5ec0b708760296902e7a6be878b8ed4dcab0928a83b07d3784826 Nov 01 22:45:43.921707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.921669 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:43.921815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.921772 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:43.921855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.921820 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:45:43.940539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.940510 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:45:43.951101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.951082 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-plugins-config-volume\") pod \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " Nov 01 22:45:43.951327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.951310 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-kube-api-access-zvn4d\" (UniqueName: \"kubernetes.io/projected/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-kube-api-access-zvn4d\") pod \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\" (UID: \"f3f42c4d-cebd-462d-a59c-4252fa0b8e78\") " Nov 01 22:45:43.951476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.951440 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-plugins-config-volume" pod "f3f42c4d-cebd-462d-a59c-4252fa0b8e78" (UID: "f3f42c4d-cebd-462d-a59c-4252fa0b8e78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:43.951952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.951926 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:43.953538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:43.953512 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-kube-api-access-zvn4d" pod "f3f42c4d-cebd-462d-a59c-4252fa0b8e78" (UID: "f3f42c4d-cebd-462d-a59c-4252fa0b8e78"). InnerVolumeSpecName "kube-api-access-zvn4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:44.053081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.053048 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvn4d\" (UniqueName: \"kubernetes.io/projected/f3f42c4d-cebd-462d-a59c-4252fa0b8e78-kube-api-access-zvn4d\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:44.378285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.378251 4981 generic.go:356] "Generic (PLEG): container finished" podID="f3f42c4d-cebd-462d-a59c-4252fa0b8e78" containerID="21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33" exitCode=0 Nov 01 22:45:44.378745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.378315 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" Nov 01 22:45:44.378745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.378337 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" event={"ID":"f3f42c4d-cebd-462d-a59c-4252fa0b8e78","Type":"ContainerDied","Data":"21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33"} Nov 01 22:45:44.378745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.378392 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl" event={"ID":"f3f42c4d-cebd-462d-a59c-4252fa0b8e78","Type":"ContainerDied","Data":"53bf4a29231965f30473f100230adf9f4aba9ffd22dce0fdd96cae18b2df39dc"} Nov 01 22:45:44.378745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.378418 4981 scope.go:122] "RemoveContainer" containerID="21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33" Nov 01 22:45:44.380685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.380659 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" event={"ID":"6c078984-19f8-4130-9ec5-6ae240e8557d","Type":"ContainerStarted","Data":"386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18"} Nov 01 22:45:44.380815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.380732 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" event={"ID":"6c078984-19f8-4130-9ec5-6ae240e8557d","Type":"ContainerStarted","Data":"f21eb61636b5ec0b708760296902e7a6be878b8ed4dcab0928a83b07d3784826"} Nov 01 22:45:44.393966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.393949 4981 scope.go:122] "RemoveContainer" containerID="21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33" Nov 01 22:45:44.394258 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:44.394240 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33\": container with ID starting with 21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33 not found: ID does not exist" containerID="21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33" Nov 01 22:45:44.394317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.394266 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33"} err="failed to get container status \"21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33\": rpc error: code = NotFound desc = could not find container \"21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33\": container with ID starting with 21a0d748195ae4cc6e2c01a7e557f6ed9c9ea1f685a3de63f9de1edd7654cf33 not found: ID does not exist" Nov 01 22:45:44.415271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.415223 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podStartSLOduration=2.415210833 podStartE2EDuration="2.415210833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:44.411714541 +0000 UTC m=+1309.839592477" watchObservedRunningTime="2026-11-01 22:45:44.415210833 +0000 UTC m=+1309.843088854" Nov 01 22:45:44.435356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.435317 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl"] Nov 01 22:45:44.440192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.440173 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-df4b43-594c88d5f9-5b5wl"] Nov 01 22:45:44.550600 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:45:44.538796 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:45:44.607603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.601855 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:44.742145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:44.742115 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f42c4d-cebd-462d-a59c-4252fa0b8e78" path="/var/lib/kubelet/pods/f3f42c4d-cebd-462d-a59c-4252fa0b8e78/volumes" Nov 01 22:45:45.047251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:45.047150 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:45.600853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:45.600824 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:45.807231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:45.807197 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:46.226444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:46.226399 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:46.402972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:46.402942 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:46.403891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:46.403871 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:45:46.771229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:46.771197 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:46.924554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:46.924519 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:47.469915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:47.469884 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:47.943486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:47.943455 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:48.151377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:48.151338 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799"] Nov 01 22:45:48.161924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:48.161888 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-l9799"] Nov 01 22:45:48.170686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:48.170636 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:48.320943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:48.320839 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:48.594737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:48.594649 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:48.826626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:48.824505 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:48.961456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:48.914464 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c" path="/var/lib/kubelet/pods/7f83d7b3-90e6-4e93-80a4-1f7c7a41bc9c/volumes" Nov 01 22:45:49.963534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:49.963502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:50.058087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:50.058058 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b"] Nov 01 22:45:50.073538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:50.073511 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00ddad3f-gbw9b"] Nov 01 22:45:50.738718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:50.738681 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9229dc2-9126-4d66-a93a-a433e6a3dcd7" path="/var/lib/kubelet/pods/c9229dc2-9126-4d66-a93a-a433e6a3dcd7/volumes" Nov 01 22:45:51.026835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:51.026738 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:52.038288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:52.038241 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:52.284484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:52.284457 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:53.100564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:53.100518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:54.222600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.212291 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l"] Nov 01 22:45:54.222600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.212542 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" containerID="cri-o://2fa7b5d3c9f255115424b50bd23e77a7f6e97e4556499d69e0cee9a71935e10a" gracePeriod=30 Nov 01 22:45:54.484674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.484571 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f"] Nov 01 22:45:54.485411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.485377 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" containerID="cri-o://f1a201036d970e1107ee0def481a18fd4a5f13a5d4c23552ce17a84bdb3af3a2" gracePeriod=30 Nov 01 22:45:54.491481 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.491458 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6"] Nov 01 22:45:54.491884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.491858 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" containerID="cri-o://39f53f2729c7afb2a223021496a5dbb442762bddcd662ed0b8d9c12a04b5c8f7" gracePeriod=30 Nov 01 22:45:54.496996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.496971 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z"] Nov 01 22:45:54.497838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.497806 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" containerID="cri-o://39b862b8137fde23d327c3f5aa77486ecec1b56deb3834e32eea43b5dfc30aab" gracePeriod=30 Nov 01 22:45:54.722259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.722172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:54.856159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:54.856079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:55.159656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.144857 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:55.362170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.361746 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:45:55.362170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.361845 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:45:55.751596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.750934 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:45:55.751596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.751006 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:45:55.831702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.831662 4981 patch_prober.go:29] interesting pod/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:45:55.832017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.831987 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:45:55.893596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:55.892152 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:56.106650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.105630 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6"] Nov 01 22:45:56.139833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.139797 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc0c68-lpqh6"] Nov 01 22:45:56.253671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.253517 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:56.648600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.644729 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:56.695601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.684690 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:45:56.718600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.714666 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:56.736597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.733081 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:56.773603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.772638 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:45:56.834224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.834181 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:56.879384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.879346 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:56.880731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.880703 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f69691-5323-400a-8021-83623c2b3c8b" path="/var/lib/kubelet/pods/c6f69691-5323-400a-8021-83623c2b3c8b/volumes" Nov 01 22:45:57.069600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.977340 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.069600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.978315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:57.069600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:56.997460 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:45:57.069600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.029215 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.069600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.029281 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.069600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.029354 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.284600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.256113 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:45:57.317706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.286729 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:45:57.337311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.337022 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.436743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.436460 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:57.695244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.678100 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.720598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.719116 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.720887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.720867 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:45:57.722469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.722446 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb"] Nov 01 22:45:57.723862 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.723837 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f3f42c4d-cebd-462d-a59c-4252fa0b8e78" containerName="payload-processing" Nov 01 22:45:57.724846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.724829 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f42c4d-cebd-462d-a59c-4252fa0b8e78" containerName="payload-processing" Nov 01 22:45:57.733625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.725203 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f42c4d-cebd-462d-a59c-4252fa0b8e78" containerName="payload-processing" Nov 01 22:45:57.754686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.734700 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:45:57.782818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.738179 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.783953 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.776613 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb"] Nov 01 22:45:57.784325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.784307 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:45:57.793100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.793064 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:45:57.795449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.776676 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:45:57.799548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.776775 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:57.799548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.797463 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:45:57.799548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.777760 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.813132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.802497 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Nov 01 22:45:57.816420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.815716 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-30578128\"" Nov 01 22:45:57.816420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.816169 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-30578128-dockercfg-hgwzc\"" Nov 01 22:45:57.853448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.836785 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:57.853448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.838123 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:45:57.913927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.913882 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6n2\" (UniqueName: \"kubernetes.io/projected/76eae60f-3b64-434d-8d16-0d9bd4412be6-kube-api-access-ld6n2\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:58.010340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:57.994629 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76eae60f-3b64-434d-8d16-0d9bd4412be6-plugins-config-volume\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:58.095126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.075568 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:45:58.095126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.076070 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:45:58.095126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.077744 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:45:58.095126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.077970 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:58.095126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.078142 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:45:58.095126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.078297 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:45:58.115782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.097521 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:45:58.115782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.097606 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:45:58.115782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.097641 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:45:58.115782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.097672 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:45:58.119609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.117491 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76eae60f-3b64-434d-8d16-0d9bd4412be6-plugins-config-volume\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:58.119609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.117885 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6n2\" (UniqueName: \"kubernetes.io/projected/76eae60f-3b64-434d-8d16-0d9bd4412be6-kube-api-access-ld6n2\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:58.119609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.119352 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76eae60f-3b64-434d-8d16-0d9bd4412be6-plugins-config-volume\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:58.268154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.268041 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6n2\" (UniqueName: \"kubernetes.io/projected/76eae60f-3b64-434d-8d16-0d9bd4412be6-kube-api-access-ld6n2\") pod \"payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:58.293830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.287727 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:58.425894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.425851 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:45:58.481872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.470247 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:45:58.510614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.507672 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:45:58.676598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.669850 4981 scope.go:122] "RemoveContainer" containerID="5507dc7fb63b7b7e23bc13c2b6cd9b39f833abcbc02f9813a5a9af6d1aa0bb5d" Nov 01 22:45:58.695950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.695569 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:45:58.701566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.701547 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:45:58.702385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.702350 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:45:58.702607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.702595 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:45:58.702775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.702760 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:45:58.702965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.702954 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:45:58.703133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.703113 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:45:58.703283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.703270 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:45:58.703431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.703419 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:45:58.703650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.703638 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:45:58.703806 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.703792 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:45:58.703962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.703950 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:45:58.704105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.704094 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:45:58.714086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.714069 4981 scope.go:122] "RemoveContainer" containerID="6e3e52b610d2f5035055fe168acecf35e50fd75746c7569addf6f13445ba32af" Nov 01 22:45:58.718504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.718487 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:45:58.791756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.791728 4981 scope.go:122] "RemoveContainer" containerID="55cdf8b2b78b029002969f1c44c0fc0a6c56c73f9550ce19fd014650983e0b4a" Nov 01 22:45:58.810238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.805118 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:45:58.820318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.812851 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:45:58.824791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.824258 4981 scope.go:122] "RemoveContainer" containerID="b3b23bac32827b37ffbf48e858276273b3b7ef2907f3c146ac69121d8d735ca8" Nov 01 22:45:58.849537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.847526 4981 scope.go:122] "RemoveContainer" containerID="584d560864863bcaf19adc7a91ad6405e2f8ca36c5632a31d139114981f3fbac" Nov 01 22:45:58.894276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.894252 4981 scope.go:122] "RemoveContainer" containerID="3dd5f244aeb02ffe52c5ac4592ac3916a1433fb389a472d0102a493b148455d8" Nov 01 22:45:58.908468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.908350 4981 scope.go:122] "RemoveContainer" containerID="7e4b991cc285adf46708ef65eb6f0cc03c66789d14f5735984f077af105ffd66" Nov 01 22:45:58.941654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.941596 4981 scope.go:122] "RemoveContainer" containerID="0960ce90f44e9ce5e6a547129385c3e2a6453cfdcd1924e8f701cf82ed7db85b" Nov 01 22:45:58.965332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.965304 4981 scope.go:122] "RemoveContainer" containerID="99e82a00787c9d41542ce95af1b6437b2fa5f6eca412af2e2eab79121dd5a833" Nov 01 22:45:58.966239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.966206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:58.978692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.978670 4981 scope.go:122] "RemoveContainer" containerID="317a714a610e0bc97c34fdd8134c9c2d47be60eafe20872398ca8d044c1da7c5" Nov 01 22:45:58.993421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.993396 4981 scope.go:122] "RemoveContainer" containerID="c5d6239fe24f368e7f9bdff9267066dbd16c9fff581a920a96feda2fa191c58c" Nov 01 22:45:58.997207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:58.997185 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb"] Nov 01 22:45:58.997384 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:45:58.997363 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eae60f_3b64_434d_8d16_0d9bd4412be6.slice/crio-588ed30b58249e2860c4c0e122cf1c94356f2ca8f836e111d5c023a38c2279b6 WatchSource:0}: Error finding container 588ed30b58249e2860c4c0e122cf1c94356f2ca8f836e111d5c023a38c2279b6: Status 404 returned error can't find the container with id 588ed30b58249e2860c4c0e122cf1c94356f2ca8f836e111d5c023a38c2279b6 Nov 01 22:45:59.029011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.028902 4981 scope.go:122] "RemoveContainer" containerID="14b2986848df9e93a99449386dfaccdbf927650927d089e9e931218fc2485145" Nov 01 22:45:59.478124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.477965 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:45:59.714382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.714206 4981 generic.go:356] "Generic (PLEG): container finished" podID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerID="2fa7b5d3c9f255115424b50bd23e77a7f6e97e4556499d69e0cee9a71935e10a" exitCode=0 Nov 01 22:45:59.714382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.714285 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" event={"ID":"21e4d4eb-eb65-47d5-ae4e-348b29afbe85","Type":"ContainerDied","Data":"2fa7b5d3c9f255115424b50bd23e77a7f6e97e4556499d69e0cee9a71935e10a"} Nov 01 22:45:59.714382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.714320 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" event={"ID":"21e4d4eb-eb65-47d5-ae4e-348b29afbe85","Type":"ContainerDied","Data":"d48f87e971f1df28883b655c54cf334371069fdc04a5adcb5a97c489e3e8c142"} Nov 01 22:45:59.714382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.714337 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d48f87e971f1df28883b655c54cf334371069fdc04a5adcb5a97c489e3e8c142" Nov 01 22:45:59.717280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.716067 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:45:59.719441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.719084 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" event={"ID":"76eae60f-3b64-434d-8d16-0d9bd4412be6","Type":"ContainerStarted","Data":"b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1"} Nov 01 22:45:59.719441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.719130 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" event={"ID":"76eae60f-3b64-434d-8d16-0d9bd4412be6","Type":"ContainerStarted","Data":"588ed30b58249e2860c4c0e122cf1c94356f2ca8f836e111d5c023a38c2279b6"} Nov 01 22:45:59.719441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.719238 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:45:59.732479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.730801 4981 generic.go:356] "Generic (PLEG): container finished" podID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerID="f1a201036d970e1107ee0def481a18fd4a5f13a5d4c23552ce17a84bdb3af3a2" exitCode=0 Nov 01 22:45:59.732479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.730879 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" event={"ID":"e8c07ded-c806-40fb-88af-7ea7c2648bdd","Type":"ContainerDied","Data":"f1a201036d970e1107ee0def481a18fd4a5f13a5d4c23552ce17a84bdb3af3a2"} Nov 01 22:45:59.734379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.734359 4981 generic.go:356] "Generic (PLEG): container finished" podID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerID="39f53f2729c7afb2a223021496a5dbb442762bddcd662ed0b8d9c12a04b5c8f7" exitCode=0 Nov 01 22:45:59.734788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.734415 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" event={"ID":"fb7f3fdd-d5d9-4c31-b98d-e37188a72983","Type":"ContainerDied","Data":"39f53f2729c7afb2a223021496a5dbb442762bddcd662ed0b8d9c12a04b5c8f7"} Nov 01 22:45:59.736622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.736457 4981 generic.go:356] "Generic (PLEG): container finished" podID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerID="39b862b8137fde23d327c3f5aa77486ecec1b56deb3834e32eea43b5dfc30aab" exitCode=0 Nov 01 22:45:59.736842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.736772 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" event={"ID":"1c72bed6-36e0-47d4-b761-53d752aa7962","Type":"ContainerDied","Data":"39b862b8137fde23d327c3f5aa77486ecec1b56deb3834e32eea43b5dfc30aab"} Nov 01 22:45:59.777713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.777660 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" podStartSLOduration=2.777560002 podStartE2EDuration="2.777560002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:59.776001803 +0000 UTC m=+1325.203879735" watchObservedRunningTime="2026-11-01 22:45:59.777560002 +0000 UTC m=+1325.205437933" Nov 01 22:45:59.792786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.791687 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-kube-api-access-v74gm\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-kube-api-access-v74gm\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.792786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.792479 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-certs" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.792786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.792538 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-certs\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.793077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.792896 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-token\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-token\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.793077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.792992 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-podinfo\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.793447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.793368 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istiod-ca-cert\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.793451 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-data\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.793617 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-envoy\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.793724 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-credential-socket\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.793781 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-socket\") pod \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\" (UID: \"21e4d4eb-eb65-47d5-ae4e-348b29afbe85\") " Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.794264 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-data" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.794549 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-socket" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.794733 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istiod-ca-cert" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:59.794793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.794493 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-credential-socket" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.799520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.798192 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.799520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.798219 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.799520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.798237 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.799520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.798250 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.799520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.798265 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.800757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.800734 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-envoy" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.819881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.809398 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-kube-api-access-v74gm" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "kube-api-access-v74gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:59.819881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.819557 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-podinfo" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:45:59.819881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.819820 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-token" pod "21e4d4eb-eb65-47d5-ae4e-348b29afbe85" (UID: "21e4d4eb-eb65-47d5-ae4e-348b29afbe85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:59.839560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.839532 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:45:59.876804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.876709 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:45:59.882722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.882695 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:45:59.899242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899217 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-socket\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899332 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-certs\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899373 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-kube-api-access-hbd2j\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-kube-api-access-hbd2j\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899416 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-token\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-token\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899449 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-socket" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.899526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899480 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-credential-socket\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899570 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-envoy\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899631 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-certs" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.899824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899775 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-data\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899910 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istiod-ca-cert\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.899989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.899952 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-podinfo\") pod \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\" (UID: \"e8c07ded-c806-40fb-88af-7ea7c2648bdd\") " Nov 01 22:45:59.900047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.900024 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-data" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.901136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.900298 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istiod-ca-cert" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:45:59.902191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902159 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-envoy" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.902459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902443 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902468 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-token" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:59.902535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902480 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902499 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902509 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v74gm\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-kube-api-access-v74gm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902517 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902526 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902536 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902544 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e4d4eb-eb65-47d5-ae4e-348b29afbe85-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:45:59.902965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902722 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-kube-api-access-hbd2j" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "kube-api-access-hbd2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:45:59.902965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.902770 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-credential-socket" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:45:59.903129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:45:59.903009 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-podinfo" pod "e8c07ded-c806-40fb-88af-7ea7c2648bdd" (UID: "e8c07ded-c806-40fb-88af-7ea7c2648bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:00.003763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003685 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-certs\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.003763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003734 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-data\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.003971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003765 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-kube-api-access-k4n6r\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-kube-api-access-k4n6r\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.003971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003788 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-envoy\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.003971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003850 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-data\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.003971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003911 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istiod-ca-cert\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.003971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003913 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-certs" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.004235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.003979 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-socket\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.004235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004005 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-token\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-token\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.004235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004038 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-data" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.004235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004101 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-credential-socket\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.004235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004204 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-data" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004247 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-podinfo\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004281 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-token\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-token\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004331 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-socket" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004352 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-envoy\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004384 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-credential-socket\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004436 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-socket\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004465 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-credential-socket" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.004498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004474 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-podinfo\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.004972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004500 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istiod-ca-cert" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:00.004972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004553 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-kube-api-access-qxxbn\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-kube-api-access-qxxbn\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.004972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004598 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-certs\") pod \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\" (UID: \"fb7f3fdd-d5d9-4c31-b98d-e37188a72983\") " Nov 01 22:46:00.004972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004671 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c72bed6-36e0-47d4-b761-53d752aa7962-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c72bed6-36e0-47d4-b761-53d752aa7962-istiod-ca-cert\") pod \"1c72bed6-36e0-47d4-b761-53d752aa7962\" (UID: \"1c72bed6-36e0-47d4-b761-53d752aa7962\") " Nov 01 22:46:00.004972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.004697 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-credential-socket" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.005690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.005666 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.005872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.005855 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.005970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.005956 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbd2j\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-kube-api-access-hbd2j\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006048 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006127 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006145 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.005731 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-certs" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.006203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006031 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c72bed6-36e0-47d4-b761-53d752aa7962-istiod-ca-cert" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:00.006203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006123 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-socket" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.006418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006206 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006220 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006234 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006248 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006262 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c07ded-c806-40fb-88af-7ea7c2648bdd-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006275 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.006828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006509 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-envoy" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.006828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006805 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-kube-api-access-k4n6r" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "kube-api-access-k4n6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:00.006951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.006884 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-podinfo" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:00.007073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.007055 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-token" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:00.007155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.007088 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-envoy" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:00.007408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.007386 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-podinfo" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:00.007977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.007958 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-kube-api-access-qxxbn" pod "1c72bed6-36e0-47d4-b761-53d752aa7962" (UID: "1c72bed6-36e0-47d4-b761-53d752aa7962"). InnerVolumeSpecName "kube-api-access-qxxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:00.008130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.008112 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-token" pod "fb7f3fdd-d5d9-4c31-b98d-e37188a72983" (UID: "fb7f3fdd-d5d9-4c31-b98d-e37188a72983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:00.107717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107681 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4n6r\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-kube-api-access-k4n6r\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107716 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107729 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107740 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107751 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107758 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107767 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c72bed6-36e0-47d4-b761-53d752aa7962-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107777 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107785 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxxbn\" (UniqueName: \"kubernetes.io/projected/1c72bed6-36e0-47d4-b761-53d752aa7962-kube-api-access-qxxbn\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107799 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb7f3fdd-d5d9-4c31-b98d-e37188a72983-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.107926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.107807 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c72bed6-36e0-47d4-b761-53d752aa7962-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:00.591303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.591265 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:00.764823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.763280 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" event={"ID":"e8c07ded-c806-40fb-88af-7ea7c2648bdd","Type":"ContainerDied","Data":"3986f8c392322ba4fe95be8af6482d84529578bdd041ca2528966e6fe123c777"} Nov 01 22:46:00.764823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.763304 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f" Nov 01 22:46:00.764823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.763373 4981 scope.go:122] "RemoveContainer" containerID="f1a201036d970e1107ee0def481a18fd4a5f13a5d4c23552ce17a84bdb3af3a2" Nov 01 22:46:00.765646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.765048 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" Nov 01 22:46:00.765646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.765163 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6" event={"ID":"fb7f3fdd-d5d9-4c31-b98d-e37188a72983","Type":"ContainerDied","Data":"8b984ecb5ab7c38d9eaf3b2381e07af4784d5189686d56da7e95f39212376d4e"} Nov 01 22:46:00.767914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.767894 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l" Nov 01 22:46:00.768176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.768155 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" Nov 01 22:46:00.768257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.768190 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z" event={"ID":"1c72bed6-36e0-47d4-b761-53d752aa7962","Type":"ContainerDied","Data":"40b720219c32bd2f028e71010d2cef14074dacf7d593c3265e51effcd9d27127"} Nov 01 22:46:00.776164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.776146 4981 scope.go:122] "RemoveContainer" containerID="39f53f2729c7afb2a223021496a5dbb442762bddcd662ed0b8d9c12a04b5c8f7" Nov 01 22:46:00.787259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.787241 4981 scope.go:122] "RemoveContainer" containerID="39b862b8137fde23d327c3f5aa77486ecec1b56deb3834e32eea43b5dfc30aab" Nov 01 22:46:00.792294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.792275 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f"] Nov 01 22:46:00.801314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.801296 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-9pb2f"] Nov 01 22:46:00.830496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.830319 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z"] Nov 01 22:46:00.835570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.835549 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-rvg9z"] Nov 01 22:46:00.852793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.852762 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l"] Nov 01 22:46:00.865734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.858873 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-8kd2l"] Nov 01 22:46:00.880945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.880917 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6"] Nov 01 22:46:00.885043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:00.885021 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-dwzq6"] Nov 01 22:46:01.416795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:01.416738 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:02.263025 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:02.262993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:02.738695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:02.738662 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" path="/var/lib/kubelet/pods/1c72bed6-36e0-47d4-b761-53d752aa7962/volumes" Nov 01 22:46:02.739237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:02.739222 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" path="/var/lib/kubelet/pods/21e4d4eb-eb65-47d5-ae4e-348b29afbe85/volumes" Nov 01 22:46:02.739700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:02.739687 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" path="/var/lib/kubelet/pods/e8c07ded-c806-40fb-88af-7ea7c2648bdd/volumes" Nov 01 22:46:02.740438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:02.740424 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" path="/var/lib/kubelet/pods/fb7f3fdd-d5d9-4c31-b98d-e37188a72983/volumes" Nov 01 22:46:03.206311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:03.206280 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:03.981986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:03.981948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:04.776702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:04.776671 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:04.972138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:04.972099 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:05.140109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:05.140016 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv"] Nov 01 22:46:05.146813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:05.145705 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df4b43-kc7wv"] Nov 01 22:46:05.326636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:05.326606 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:05.506816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:05.506778 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:06.290999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:06.290960 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:06.738050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:06.738017 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9836a54-5722-455f-9e92-2a085f4a2c3b" path="/var/lib/kubelet/pods/b9836a54-5722-455f-9e92-2a085f4a2c3b/volumes" Nov 01 22:46:07.037355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:07.037280 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:07.186693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:07.184714 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:08.069411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:08.069377 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:09.141787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:09.141743 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:09.248156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:09.248120 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:09.519355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:09.519322 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:10.234267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:10.234237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:10.771294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:10.771269 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:46:11.100395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:11.100300 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:12.068961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:12.068922 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:12.874005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:12.873970 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:12.983028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:12.982999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:13.568396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:13.568360 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:13.682346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:13.682316 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:13.826501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:13.826404 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:14.028464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:14.028425 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd"] Nov 01 22:46:14.029012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:14.028978 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" containerID="cri-o://7e5a1789f54c21eb401ae96d9c50c48296bb04e88102e949241286082c6f762a" gracePeriod=30 Nov 01 22:46:14.178148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:14.178116 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:14.528198 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:46:14.528112 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:46:14.848377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:14.848298 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:14.985266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:14.985230 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:15.268697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.268661 4981 patch_prober.go:29] interesting pod/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:46:15.268872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.268732 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:46:15.270389 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.270360 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6"] Nov 01 22:46:15.271676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271650 4981 cpu_manager.go:414] "containerMap: removing container" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" Nov 01 22:46:15.271787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271681 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" Nov 01 22:46:15.271787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271743 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" Nov 01 22:46:15.271787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271751 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" Nov 01 22:46:15.271787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271764 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" Nov 01 22:46:15.271787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271773 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" Nov 01 22:46:15.271787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271785 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" Nov 01 22:46:15.272104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.271792 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" Nov 01 22:46:15.272104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.272004 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c07ded-c806-40fb-88af-7ea7c2648bdd" containerName="istio-proxy" Nov 01 22:46:15.272104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.272020 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb7f3fdd-d5d9-4c31-b98d-e37188a72983" containerName="istio-proxy" Nov 01 22:46:15.272104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.272031 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c72bed6-36e0-47d4-b761-53d752aa7962" containerName="istio-proxy" Nov 01 22:46:15.272104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.272053 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e4d4eb-eb65-47d5-ae4e-348b29afbe85" containerName="istio-proxy" Nov 01 22:46:15.276758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.276733 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.286693 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.286672 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1799f8-openshift-default-dockercfg-fvhdd\"" Nov 01 22:46:15.296392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.296368 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6"] Nov 01 22:46:15.343069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.343040 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:15.427917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.427872 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.446596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.428362 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.446903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.446877 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csjwd\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-kube-api-access-csjwd\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.447021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.447006 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.447176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.447163 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcddcb52-6b43-4652-8057-9f898efb0a41-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.447265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.447254 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.471686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.471648 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.471995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.471973 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.472191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.472175 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.580983 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581053 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csjwd\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-kube-api-access-csjwd\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581083 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581163 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcddcb52-6b43-4652-8057-9f898efb0a41-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581190 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581211 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581278 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581321 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.582642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.581611 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.583280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.583055 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-data\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.583385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.583342 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-certs\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.584307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.584277 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-credential-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.588771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.585179 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-socket\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.589011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.585322 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcddcb52-6b43-4652-8057-9f898efb0a41-istiod-ca-cert\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.598371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.598338 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-podinfo\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.601285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.601257 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-token\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.603542 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.603515 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csjwd\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-kube-api-access-csjwd\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.611235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.605800 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-envoy\") pod \"e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:15.883116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.883039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:15.892342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:15.892305 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:16.243726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.243534 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6"] Nov 01 22:46:16.245086 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:16.244969 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcddcb52_6b43_4652_8057_9f898efb0a41.slice/crio-603b6a32a9632cddc8cbe3db22678419ed9263d6b744a6c76ce860516ac816de WatchSource:0}: Error finding container 603b6a32a9632cddc8cbe3db22678419ed9263d6b744a6c76ce860516ac816de: Status 404 returned error can't find the container with id 603b6a32a9632cddc8cbe3db22678419ed9263d6b744a6c76ce860516ac816de Nov 01 22:46:16.250936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.249487 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:16.250936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.249567 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:16.250936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.249649 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:16.548261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.548184 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:16.819124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.819047 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:16.963440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.963405 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" event={"ID":"bcddcb52-6b43-4652-8057-9f898efb0a41","Type":"ContainerStarted","Data":"fb12f37016e03f205114c0f4b7aeed2f215d69516fae46772910ca9b68562acc"} Nov 01 22:46:16.963828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.963462 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" event={"ID":"bcddcb52-6b43-4652-8057-9f898efb0a41","Type":"ContainerStarted","Data":"603b6a32a9632cddc8cbe3db22678419ed9263d6b744a6c76ce860516ac816de"} Nov 01 22:46:16.995377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:16.995324 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podStartSLOduration=1.995309497 podStartE2EDuration="1.995309497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:16.990458335 +0000 UTC m=+1342.418336263" watchObservedRunningTime="2026-11-01 22:46:16.995309497 +0000 UTC m=+1342.423187427" Nov 01 22:46:17.463766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.463731 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:17.577484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.577450 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:17.855090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.855018 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg"] Nov 01 22:46:17.859823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.859806 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:17.864358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.864328 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c0d076-openshift-default-dockercfg-62wbd\"" Nov 01 22:46:17.891029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.891005 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg"] Nov 01 22:46:17.893642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.893616 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:17.900345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:17.900320 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:18.046606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.027144 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:18.194434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194053 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-socket\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194252 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-data\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194285 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-envoy\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194396 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-token\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194464 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmgj\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-kube-api-access-crmgj\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194525 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-credential-socket\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194547 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-podinfo\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194654 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-certs\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.194815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.194772 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istiod-ca-cert\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.201385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.201360 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:18.202444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.202424 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:46:18.288350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.288309 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:18.296158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296128 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-data\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.296388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296366 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-envoy\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.296695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296672 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-data\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.296788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296676 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-token\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.296842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296800 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmgj\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-kube-api-access-crmgj\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.296899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296882 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-credential-socket\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.296954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296910 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-podinfo\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.297070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.296989 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-certs\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.297122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.297104 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istiod-ca-cert\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.297281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.297260 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-socket\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.298183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.298155 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-certs\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.298514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.298477 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-credential-socket\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.298765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.298742 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-socket\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.298892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.298818 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istiod-ca-cert\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.300625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.300599 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-envoy\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.302524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.302500 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-podinfo\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.306474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.306453 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-token\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.308070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.308049 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmgj\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-kube-api-access-crmgj\") pod \"e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.475873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.475783 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:18.766875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.766200 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg"] Nov 01 22:46:18.770788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.770753 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:18.770924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.770818 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:18.770924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.770853 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:18.934149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:18.934112 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:19.213942 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.213879 4981 generic.go:356] "Generic (PLEG): container finished" podID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerID="7e5a1789f54c21eb401ae96d9c50c48296bb04e88102e949241286082c6f762a" exitCode=0 Nov 01 22:46:19.214449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.213947 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" event={"ID":"1bef9cd2-8fd6-42e4-865c-0004f50dd88a","Type":"ContainerDied","Data":"7e5a1789f54c21eb401ae96d9c50c48296bb04e88102e949241286082c6f762a"} Nov 01 22:46:19.214449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.214010 4981 scope.go:122] "RemoveContainer" containerID="10f7f23bf0a0ecbc00644104539551629839257ee93574c9504540714f59cb7f" Nov 01 22:46:19.217316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.217292 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" event={"ID":"8f2dd39f-6f0f-4481-b36d-6d059a93958c","Type":"ContainerStarted","Data":"cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824"} Nov 01 22:46:19.217457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.217344 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" event={"ID":"8f2dd39f-6f0f-4481-b36d-6d059a93958c","Type":"ContainerStarted","Data":"21fd70c0061ceee0912536ed779b7e043653e14f4ca0cf7d84c41ed09b18fe19"} Nov 01 22:46:19.244947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.244897 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podStartSLOduration=2.244880742 podStartE2EDuration="2.244880742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:19.239153113 +0000 UTC m=+1344.667031042" watchObservedRunningTime="2026-11-01 22:46:19.244880742 +0000 UTC m=+1344.672758673" Nov 01 22:46:19.311350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.311260 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w"] Nov 01 22:46:19.326416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.326396 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w"] Nov 01 22:46:19.326529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.326477 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.330838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.330814 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:19.331003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.330888 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:19.331128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.331104 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:19.331184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.331135 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:46:19.427026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.426450 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26w9\" (UniqueName: \"kubernetes.io/projected/5147cebe-ac7c-44d9-a93f-24c58cffee67-kube-api-access-s26w9\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.427026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.426770 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.427026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.426857 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.456193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.455415 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:19.476497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.476467 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:19.482283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.482259 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:19.528654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.528617 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.528945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.528729 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.528945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.528866 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26w9\" (UniqueName: \"kubernetes.io/projected/5147cebe-ac7c-44d9-a93f-24c58cffee67-kube-api-access-s26w9\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.529530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.529504 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.531220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.531195 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.551835 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.551802 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26w9\" (UniqueName: \"kubernetes.io/projected/5147cebe-ac7c-44d9-a93f-24c58cffee67-kube-api-access-s26w9\") pod \"maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.578055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.577981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:19.645635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.645605 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:19.695492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.695460 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:19.737795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.737699 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:46:19.841319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841236 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-credential-socket\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.841489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841411 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-envoy\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.841489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841458 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-token\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-token\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.841619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841530 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-credential-socket" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:19.841671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841653 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-podinfo\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.841720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841704 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-data\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.841871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841841 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-kube-api-access-ml298\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-kube-api-access-ml298\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.842022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841910 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-socket\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.842022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841936 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-data" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:19.842022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.841985 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istiod-ca-cert\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.842186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.842109 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-certs\") pod \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\" (UID: \"1bef9cd2-8fd6-42e4-865c-0004f50dd88a\") " Nov 01 22:46:19.842475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.842447 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-socket" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:19.842708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.842686 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istiod-ca-cert" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:19.842933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.842914 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-certs" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:19.843458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.843435 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.843562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.843460 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.843562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.843474 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.843562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.843489 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.843562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.843502 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.843968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.843948 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-token" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:19.844172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.844149 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-envoy" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:46:19.844213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.844198 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-kube-api-access-ml298" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "kube-api-access-ml298". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:19.845062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.845044 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-podinfo" pod "1bef9cd2-8fd6-42e4-865c-0004f50dd88a" (UID: "1bef9cd2-8fd6-42e4-865c-0004f50dd88a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:46:19.945284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.945243 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.945284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.945272 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml298\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-kube-api-access-ml298\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.945284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.945282 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.945284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.945290 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bef9cd2-8fd6-42e4-865c-0004f50dd88a-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:19.989716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:19.989687 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w"] Nov 01 22:46:19.990097 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:19.990070 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5147cebe_ac7c_44d9_a93f_24c58cffee67.slice/crio-ac8a912f335809517c8083b7cc7009629fe280ebecb7d62cce5375430a292805 WatchSource:0}: Error finding container ac8a912f335809517c8083b7cc7009629fe280ebecb7d62cce5375430a292805: Status 404 returned error can't find the container with id ac8a912f335809517c8083b7cc7009629fe280ebecb7d62cce5375430a292805 Nov 01 22:46:20.025699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.025669 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:20.228809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.228772 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" event={"ID":"5147cebe-ac7c-44d9-a93f-24c58cffee67","Type":"ContainerStarted","Data":"ac8a912f335809517c8083b7cc7009629fe280ebecb7d62cce5375430a292805"} Nov 01 22:46:20.230188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.230159 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" Nov 01 22:46:20.230309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.230184 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd" event={"ID":"1bef9cd2-8fd6-42e4-865c-0004f50dd88a","Type":"ContainerDied","Data":"b85412c149c75f2730c419833d5c06e0d044d3c045bdf74b2a77cb3e3953ab33"} Nov 01 22:46:20.230309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.230241 4981 scope.go:122] "RemoveContainer" containerID="7e5a1789f54c21eb401ae96d9c50c48296bb04e88102e949241286082c6f762a" Nov 01 22:46:20.230788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.230763 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:20.231792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.231769 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:46:20.302665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.302636 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd"] Nov 01 22:46:20.309992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.309958 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-df4b43-openshift-default-7bb7689f8f-q6wmd"] Nov 01 22:46:20.740518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.740440 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" path="/var/lib/kubelet/pods/1bef9cd2-8fd6-42e4-865c-0004f50dd88a/volumes" Nov 01 22:46:20.825289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.825255 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:20.890705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.890674 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7"] Nov 01 22:46:20.891834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.891804 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" Nov 01 22:46:20.891834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.891830 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" Nov 01 22:46:20.892034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.891880 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" Nov 01 22:46:20.892034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.891886 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" Nov 01 22:46:20.892034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.892023 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" Nov 01 22:46:20.892034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.892036 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bef9cd2-8fd6-42e4-865c-0004f50dd88a" containerName="istio-proxy" Nov 01 22:46:20.896384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.896362 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:20.910064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.910035 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7"] Nov 01 22:46:20.983768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.983739 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1799f8-dockercfg-dnstb\"" Nov 01 22:46:20.983927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.983782 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1799f8\"" Nov 01 22:46:20.993004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.992930 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cabded-6252-4cb6-95c8-f0ec9024a589-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:20.993146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:20.993107 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbgj5\" (UniqueName: \"kubernetes.io/projected/a8cabded-6252-4cb6-95c8-f0ec9024a589-kube-api-access-kbgj5\") pod \"payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:21.087470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.080158 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:21.124599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.096910 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cabded-6252-4cb6-95c8-f0ec9024a589-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:21.124599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.097064 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbgj5\" (UniqueName: \"kubernetes.io/projected/a8cabded-6252-4cb6-95c8-f0ec9024a589-kube-api-access-kbgj5\") pod \"payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:21.124599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.098696 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cabded-6252-4cb6-95c8-f0ec9024a589-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:21.145644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.140236 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbgj5\" (UniqueName: \"kubernetes.io/projected/a8cabded-6252-4cb6-95c8-f0ec9024a589-kube-api-access-kbgj5\") pod \"payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:21.231514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.231478 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:21.347832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.347805 4981 generic.go:356] "Generic (PLEG): container finished" podID="5147cebe-ac7c-44d9-a93f-24c58cffee67" containerID="51c1504e85fedafcf9d9100c33b1c262a1159ab130d2fd40c028f1ac9d522877" exitCode=0 Nov 01 22:46:21.348748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.347926 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" event={"ID":"5147cebe-ac7c-44d9-a93f-24c58cffee67","Type":"ContainerDied","Data":"51c1504e85fedafcf9d9100c33b1c262a1159ab130d2fd40c028f1ac9d522877"} Nov 01 22:46:21.461833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.461791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:21.631746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.631570 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7"] Nov 01 22:46:21.635527 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:21.635500 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cabded_6252_4cb6_95c8_f0ec9024a589.slice/crio-60652d169627bd150459eaddbaf16afdc9d8f52ebc57cbb7c55ebb14aef24872 WatchSource:0}: Error finding container 60652d169627bd150459eaddbaf16afdc9d8f52ebc57cbb7c55ebb14aef24872: Status 404 returned error can't find the container with id 60652d169627bd150459eaddbaf16afdc9d8f52ebc57cbb7c55ebb14aef24872 Nov 01 22:46:21.691208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.691178 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:21.883630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:21.883512 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:22.238470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.238432 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:22.360526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.360493 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" event={"ID":"a8cabded-6252-4cb6-95c8-f0ec9024a589","Type":"ContainerStarted","Data":"a183bfad774e2a93eb57f11966efdeff1fbeec862f7687c52524f0ef44df5a49"} Nov 01 22:46:22.360732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.360550 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" event={"ID":"a8cabded-6252-4cb6-95c8-f0ec9024a589","Type":"ContainerStarted","Data":"60652d169627bd150459eaddbaf16afdc9d8f52ebc57cbb7c55ebb14aef24872"} Nov 01 22:46:22.389011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.388962 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" podStartSLOduration=2.388949514 podStartE2EDuration="2.388949514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:22.387688895 +0000 UTC m=+1347.815566826" watchObservedRunningTime="2026-11-01 22:46:22.388949514 +0000 UTC m=+1347.816827445" Nov 01 22:46:22.582048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.582011 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:22.751785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.751756 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-openshift-ca\") pod \"5147cebe-ac7c-44d9-a93f-24c58cffee67\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " Nov 01 22:46:22.751960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.751854 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5147cebe-ac7c-44d9-a93f-24c58cffee67-kube-api-access-s26w9\" (UniqueName: \"kubernetes.io/projected/5147cebe-ac7c-44d9-a93f-24c58cffee67-kube-api-access-s26w9\") pod \"5147cebe-ac7c-44d9-a93f-24c58cffee67\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " Nov 01 22:46:22.752005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.751989 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-xks-ca\") pod \"5147cebe-ac7c-44d9-a93f-24c58cffee67\" (UID: \"5147cebe-ac7c-44d9-a93f-24c58cffee67\") " Nov 01 22:46:22.752186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.752165 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-openshift-ca" pod "5147cebe-ac7c-44d9-a93f-24c58cffee67" (UID: "5147cebe-ac7c-44d9-a93f-24c58cffee67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:22.752654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.752633 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:22.753981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.753959 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-xks-ca" pod "5147cebe-ac7c-44d9-a93f-24c58cffee67" (UID: "5147cebe-ac7c-44d9-a93f-24c58cffee67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:22.754050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.753989 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5147cebe-ac7c-44d9-a93f-24c58cffee67-kube-api-access-s26w9" pod "5147cebe-ac7c-44d9-a93f-24c58cffee67" (UID: "5147cebe-ac7c-44d9-a93f-24c58cffee67"). InnerVolumeSpecName "kube-api-access-s26w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:22.855695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.855656 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s26w9\" (UniqueName: \"kubernetes.io/projected/5147cebe-ac7c-44d9-a93f-24c58cffee67-kube-api-access-s26w9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:22.855695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.855693 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5147cebe-ac7c-44d9-a93f-24c58cffee67-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:22.914028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:22.913963 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:23.060632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.060544 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4"] Nov 01 22:46:23.061441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.061423 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5147cebe-ac7c-44d9-a93f-24c58cffee67" containerName="revoke-keys" Nov 01 22:46:23.061493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.061443 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5147cebe-ac7c-44d9-a93f-24c58cffee67" containerName="revoke-keys" Nov 01 22:46:23.061651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.061639 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5147cebe-ac7c-44d9-a93f-24c58cffee67" containerName="revoke-keys" Nov 01 22:46:23.065471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.065449 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk"] Nov 01 22:46:23.065549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.065475 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.070137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.070113 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.095734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.095707 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c0d076\"" Nov 01 22:46:23.095957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.095942 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c0d076-dockercfg-fl8bq\"" Nov 01 22:46:23.110253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.110218 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4"] Nov 01 22:46:23.115951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.115918 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk"] Nov 01 22:46:23.244596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.243738 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2b93ab-425b-4070-b572-93e16f27f2e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.244596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.243831 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.244596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.244101 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2nr\" (UniqueName: \"kubernetes.io/projected/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-kube-api-access-tn2nr\") pod \"payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.244596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.244231 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qt4\" (UniqueName: \"kubernetes.io/projected/2a2b93ab-425b-4070-b572-93e16f27f2e8-kube-api-access-k7qt4\") pod \"payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.334709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.279691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:23.380616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.378536 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2b93ab-425b-4070-b572-93e16f27f2e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.439604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.430934 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.439604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.431332 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2nr\" (UniqueName: \"kubernetes.io/projected/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-kube-api-access-tn2nr\") pod \"payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.439604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.431507 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qt4\" (UniqueName: \"kubernetes.io/projected/2a2b93ab-425b-4070-b572-93e16f27f2e8-kube-api-access-k7qt4\") pod \"payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.439604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.433301 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.439604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.380199 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2b93ab-425b-4070-b572-93e16f27f2e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.465624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.463121 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qt4\" (UniqueName: \"kubernetes.io/projected/2a2b93ab-425b-4070-b572-93e16f27f2e8-kube-api-access-k7qt4\") pod \"payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.477040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.473511 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" Nov 01 22:46:23.477040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.474973 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2nr\" (UniqueName: \"kubernetes.io/projected/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-kube-api-access-tn2nr\") pod \"payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.477040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.475022 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w" event={"ID":"5147cebe-ac7c-44d9-a93f-24c58cffee67","Type":"ContainerDied","Data":"ac8a912f335809517c8083b7cc7009629fe280ebecb7d62cce5375430a292805"} Nov 01 22:46:23.477040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.475086 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac8a912f335809517c8083b7cc7009629fe280ebecb7d62cce5375430a292805" Nov 01 22:46:23.477040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.475484 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:23.683976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.683944 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:23.693136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.690736 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:23.700112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:23.700089 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:24.080521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.080493 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk"] Nov 01 22:46:24.080833 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:24.080814 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2b93ab_425b_4070_b572_93e16f27f2e8.slice/crio-a1d68d79c8059f0d40fc32fee6be032e678a7d275f8198a57a74c83a71812f6c WatchSource:0}: Error finding container a1d68d79c8059f0d40fc32fee6be032e678a7d275f8198a57a74c83a71812f6c: Status 404 returned error can't find the container with id a1d68d79c8059f0d40fc32fee6be032e678a7d275f8198a57a74c83a71812f6c Nov 01 22:46:24.082422 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:24.082397 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814ffa64_0d94_4ace_8fc3_0ce77cecb94b.slice/crio-30c5c3363c58d4b0fef1d03f96d9feefd812c6a04fd0333e24fb38d2468fc064 WatchSource:0}: Error finding container 30c5c3363c58d4b0fef1d03f96d9feefd812c6a04fd0333e24fb38d2468fc064: Status 404 returned error can't find the container with id 30c5c3363c58d4b0fef1d03f96d9feefd812c6a04fd0333e24fb38d2468fc064 Nov 01 22:46:24.087191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.087165 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4"] Nov 01 22:46:24.483981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.483944 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" event={"ID":"814ffa64-0d94-4ace-8fc3-0ce77cecb94b","Type":"ContainerStarted","Data":"15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d"} Nov 01 22:46:24.484382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.484007 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:24.484382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.484030 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" event={"ID":"814ffa64-0d94-4ace-8fc3-0ce77cecb94b","Type":"ContainerStarted","Data":"30c5c3363c58d4b0fef1d03f96d9feefd812c6a04fd0333e24fb38d2468fc064"} Nov 01 22:46:24.485905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.485877 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" event={"ID":"2a2b93ab-425b-4070-b572-93e16f27f2e8","Type":"ContainerStarted","Data":"a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d"} Nov 01 22:46:24.486013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.485921 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" event={"ID":"2a2b93ab-425b-4070-b572-93e16f27f2e8","Type":"ContainerStarted","Data":"a1d68d79c8059f0d40fc32fee6be032e678a7d275f8198a57a74c83a71812f6c"} Nov 01 22:46:24.486053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.486023 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:24.508663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.508594 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" podStartSLOduration=2.508556554 podStartE2EDuration="2.508556554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:24.504335529 +0000 UTC m=+1349.932213462" watchObservedRunningTime="2026-11-01 22:46:24.508556554 +0000 UTC m=+1349.936434487" Nov 01 22:46:24.533284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.533235 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" podStartSLOduration=2.533222167 podStartE2EDuration="2.533222167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:24.529131117 +0000 UTC m=+1349.957009051" watchObservedRunningTime="2026-11-01 22:46:24.533222167 +0000 UTC m=+1349.961100098" Nov 01 22:46:24.663663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:24.663630 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:25.581636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:25.581587 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:26.295692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:26.295655 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:27.347093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:27.347055 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:28.182165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:28.182138 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:29.306294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:29.306262 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:29.485735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:29.485699 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6"] Nov 01 22:46:29.560723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:29.557568 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:29.656940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:29.656912 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:30.470333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:30.470301 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:31.245884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:31.245849 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:31.996280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:31.996248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:32.471534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:32.471499 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:32.783018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:32.782934 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:32.930491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:32.930458 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:33.357478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:33.357448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:33.494566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:33.494529 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:34.002419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:34.002386 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:34.487045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:34.486960 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" podUID="9f2cda48-24e5-4231-9e7a-67866b18b48b" containerName="payload-pre-processing" containerID="cri-o://eef8693a0fcd1fab0533b3b08f4cd4c23afec5fc103afad53e65ffc261c1e925" gracePeriod=25 Nov 01 22:46:34.490024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:34.490000 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:46:34.673019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:34.672986 4981 generic.go:356] "Generic (PLEG): container finished" podID="9f2cda48-24e5-4231-9e7a-67866b18b48b" containerID="eef8693a0fcd1fab0533b3b08f4cd4c23afec5fc103afad53e65ffc261c1e925" exitCode=0 Nov 01 22:46:34.673204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:34.673030 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" event={"ID":"9f2cda48-24e5-4231-9e7a-67866b18b48b","Type":"ContainerDied","Data":"eef8693a0fcd1fab0533b3b08f4cd4c23afec5fc103afad53e65ffc261c1e925"} Nov 01 22:46:34.702198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:34.702166 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:34.850063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:34.850038 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:46:35.051654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.023826 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2cda48-24e5-4231-9e7a-67866b18b48b-kube-api-access-hdbwf\" (UniqueName: \"kubernetes.io/projected/9f2cda48-24e5-4231-9e7a-67866b18b48b-kube-api-access-hdbwf\") pod \"9f2cda48-24e5-4231-9e7a-67866b18b48b\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " Nov 01 22:46:35.051654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.023974 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2cda48-24e5-4231-9e7a-67866b18b48b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cda48-24e5-4231-9e7a-67866b18b48b-plugins-config-volume\") pod \"9f2cda48-24e5-4231-9e7a-67866b18b48b\" (UID: \"9f2cda48-24e5-4231-9e7a-67866b18b48b\") " Nov 01 22:46:35.051654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.025087 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2cda48-24e5-4231-9e7a-67866b18b48b-plugins-config-volume" pod "9f2cda48-24e5-4231-9e7a-67866b18b48b" (UID: "9f2cda48-24e5-4231-9e7a-67866b18b48b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:35.100597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.093194 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2cda48-24e5-4231-9e7a-67866b18b48b-kube-api-access-hdbwf" pod "9f2cda48-24e5-4231-9e7a-67866b18b48b" (UID: "9f2cda48-24e5-4231-9e7a-67866b18b48b"). InnerVolumeSpecName "kube-api-access-hdbwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:35.147283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.127027 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdbwf\" (UniqueName: \"kubernetes.io/projected/9f2cda48-24e5-4231-9e7a-67866b18b48b-kube-api-access-hdbwf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.147283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.127064 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cda48-24e5-4231-9e7a-67866b18b48b-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:35.203602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.198699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:35.381587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.381478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:35.497426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.497396 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:46:35.497927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.497904 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:46:35.516694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.515341 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w"] Nov 01 22:46:35.524505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.523890 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a5c3234-x8q5w"] Nov 01 22:46:35.572001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.571961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:35.687334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.687294 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" Nov 01 22:46:35.687334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.687314 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6" event={"ID":"9f2cda48-24e5-4231-9e7a-67866b18b48b","Type":"ContainerDied","Data":"879c32e6ce912df119aa3d453c484d364bf8f85b56604da2bce28ec43620c5b2"} Nov 01 22:46:35.687667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.687384 4981 scope.go:122] "RemoveContainer" containerID="eef8693a0fcd1fab0533b3b08f4cd4c23afec5fc103afad53e65ffc261c1e925" Nov 01 22:46:35.750206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.750172 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6"] Nov 01 22:46:35.757748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:35.757700 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2a5c3234-6b6f84c467-xgbf6"] Nov 01 22:46:36.041820 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.041736 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:36.473819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.473776 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57"] Nov 01 22:46:36.474688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.474671 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9f2cda48-24e5-4231-9e7a-67866b18b48b" containerName="payload-pre-processing" Nov 01 22:46:36.474740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.474690 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2cda48-24e5-4231-9e7a-67866b18b48b" containerName="payload-pre-processing" Nov 01 22:46:36.474817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.474808 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2cda48-24e5-4231-9e7a-67866b18b48b" containerName="payload-pre-processing" Nov 01 22:46:36.480234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.480205 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.486989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.486962 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:46:36.486989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.486976 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:46:36.503976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.503944 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57"] Nov 01 22:46:36.509892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.509862 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:46:36.510091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.510070 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:46:36.603106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.603062 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.603318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.603177 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmq6\" (UniqueName: \"kubernetes.io/projected/0964fac4-218a-4183-851f-77b27741786c-kube-api-access-wjmq6\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.603318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.603293 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0964fac4-218a-4183-851f-77b27741786c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.717624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.716075 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.717624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.716186 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmq6\" (UniqueName: \"kubernetes.io/projected/0964fac4-218a-4183-851f-77b27741786c-kube-api-access-wjmq6\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.717624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.716297 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0964fac4-218a-4183-851f-77b27741786c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.717624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.717233 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0964fac4-218a-4183-851f-77b27741786c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.735268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.735185 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.784624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.781416 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmq6\" (UniqueName: \"kubernetes.io/projected/0964fac4-218a-4183-851f-77b27741786c-kube-api-access-wjmq6\") pod \"maas-api-revoke-keys-e2e-mt-30578128-bpl57\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.808100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.808061 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:36.836207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.825539 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5147cebe-ac7c-44d9-a93f-24c58cffee67" path="/var/lib/kubelet/pods/5147cebe-ac7c-44d9-a93f-24c58cffee67/volumes" Nov 01 22:46:36.836207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:36.826211 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2cda48-24e5-4231-9e7a-67866b18b48b" path="/var/lib/kubelet/pods/9f2cda48-24e5-4231-9e7a-67866b18b48b/volumes" Nov 01 22:46:37.179527 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:37.179490 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0964fac4_218a_4183_851f_77b27741786c.slice/crio-2929bab081f4b668838d0c48103a4be94f58ccc355bdc511c3f4ca515b63a900 WatchSource:0}: Error finding container 2929bab081f4b668838d0c48103a4be94f58ccc355bdc511c3f4ca515b63a900: Status 404 returned error can't find the container with id 2929bab081f4b668838d0c48103a4be94f58ccc355bdc511c3f4ca515b63a900 Nov 01 22:46:37.180798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:37.180773 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57"] Nov 01 22:46:37.319702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:37.319665 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:37.478561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:37.478476 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:37.873754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:37.873659 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" event={"ID":"0964fac4-218a-4183-851f-77b27741786c","Type":"ContainerStarted","Data":"ae2bb003b0e28f6bdbdbac7fce6e4c484ac77b8011bb7c2f3ad3eafcf46be6df"} Nov 01 22:46:37.875653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:37.875620 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" event={"ID":"0964fac4-218a-4183-851f-77b27741786c","Type":"ContainerStarted","Data":"2929bab081f4b668838d0c48103a4be94f58ccc355bdc511c3f4ca515b63a900"} Nov 01 22:46:37.985952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:37.978952 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" podStartSLOduration=1.97893092 podStartE2EDuration="1.97893092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:37.919154692 +0000 UTC m=+1363.347032618" watchObservedRunningTime="2026-11-01 22:46:37.97893092 +0000 UTC m=+1363.406808854" Nov 01 22:46:38.022126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:38.019827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:38.200599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:38.194008 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:39.062991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:39.062955 4981 generic.go:356] "Generic (PLEG): container finished" podID="0964fac4-218a-4183-851f-77b27741786c" containerID="ae2bb003b0e28f6bdbdbac7fce6e4c484ac77b8011bb7c2f3ad3eafcf46be6df" exitCode=0 Nov 01 22:46:39.063530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:39.063138 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" event={"ID":"0964fac4-218a-4183-851f-77b27741786c","Type":"ContainerDied","Data":"ae2bb003b0e28f6bdbdbac7fce6e4c484ac77b8011bb7c2f3ad3eafcf46be6df"} Nov 01 22:46:39.414724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:39.414642 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:39.612458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:39.612394 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:40.643440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.643350 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:40.652675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.652650 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:40.673276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.673241 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\") pod \"0964fac4-218a-4183-851f-77b27741786c\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " Nov 01 22:46:40.673567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.673549 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0964fac4-218a-4183-851f-77b27741786c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0964fac4-218a-4183-851f-77b27741786c-maas-api-openshift-ca\") pod \"0964fac4-218a-4183-851f-77b27741786c\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " Nov 01 22:46:40.673778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.673758 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0964fac4-218a-4183-851f-77b27741786c-kube-api-access-wjmq6\" (UniqueName: \"kubernetes.io/projected/0964fac4-218a-4183-851f-77b27741786c-kube-api-access-wjmq6\") pod \"0964fac4-218a-4183-851f-77b27741786c\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " Nov 01 22:46:40.703941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.703056 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0964fac4-218a-4183-851f-77b27741786c-maas-api-openshift-ca" pod "0964fac4-218a-4183-851f-77b27741786c" (UID: "0964fac4-218a-4183-851f-77b27741786c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:40.830681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.828897 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca" pod "0964fac4-218a-4183-851f-77b27741786c" (UID: "0964fac4-218a-4183-851f-77b27741786c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:40.861227 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.861187 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\") pod \"0964fac4-218a-4183-851f-77b27741786c\" (UID: \"0964fac4-218a-4183-851f-77b27741786c\") " Nov 01 22:46:40.861781 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:40.861372 4981 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/0964fac4-218a-4183-851f-77b27741786c/volumes/kubernetes.io~secret/maas-api-xks-ca Nov 01 22:46:40.861781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.861527 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca" pod "0964fac4-218a-4183-851f-77b27741786c" (UID: "0964fac4-218a-4183-851f-77b27741786c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:46:40.862927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.862906 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0964fac4-218a-4183-851f-77b27741786c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.863055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.863040 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0964fac4-218a-4183-851f-77b27741786c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:40.867810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.867774 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0964fac4-218a-4183-851f-77b27741786c-kube-api-access-wjmq6" pod "0964fac4-218a-4183-851f-77b27741786c" (UID: "0964fac4-218a-4183-851f-77b27741786c"). InnerVolumeSpecName "kube-api-access-wjmq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:40.964756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:40.964718 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjmq6\" (UniqueName: \"kubernetes.io/projected/0964fac4-218a-4183-851f-77b27741786c-kube-api-access-wjmq6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:41.132512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:41.132453 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" event={"ID":"0964fac4-218a-4183-851f-77b27741786c","Type":"ContainerDied","Data":"2929bab081f4b668838d0c48103a4be94f58ccc355bdc511c3f4ca515b63a900"} Nov 01 22:46:41.132512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:41.132503 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2929bab081f4b668838d0c48103a4be94f58ccc355bdc511c3f4ca515b63a900" Nov 01 22:46:41.132747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:41.132556 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57" Nov 01 22:46:41.757011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:41.756899 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:41.887918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:41.885091 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:42.725068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:42.725034 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:43.381559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:43.381504 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:43.492605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:43.483296 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:43.595596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:43.590793 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:43.970431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:43.970392 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:44.519566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.519531 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:44.550186 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:46:44.550152 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:46:44.647323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.647293 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:44.926652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.924671 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl"] Nov 01 22:46:44.926652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.926058 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0964fac4-218a-4183-851f-77b27741786c" containerName="revoke-keys" Nov 01 22:46:44.926652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.926081 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0964fac4-218a-4183-851f-77b27741786c" containerName="revoke-keys" Nov 01 22:46:44.926652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.926272 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0964fac4-218a-4183-851f-77b27741786c" containerName="revoke-keys" Nov 01 22:46:44.934601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.930663 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:44.936623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.934881 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:44.936623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.935158 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl"] Nov 01 22:46:44.942177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:44.941562 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b34688-openshift-default-dockercfg-8ltnl\"" Nov 01 22:46:45.027213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.027175 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114151 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-data\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114212 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istiod-ca-cert\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114290 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-socket\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114374 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jn6\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-kube-api-access-79jn6\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114488 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-certs\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114513 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-envoy\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114647 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-token\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114780 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-podinfo\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.116027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.114858 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-credential-socket\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.225679 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-podinfo\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.225790 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-credential-socket\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.225862 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-data\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.225892 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istiod-ca-cert\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.225966 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-socket\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.226049 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jn6\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-kube-api-access-79jn6\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.226157 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-certs\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.226185 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-envoy\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.226300 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-token\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.226334 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-credential-socket\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.226625 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-data\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.226645 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-socket\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.227269 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istiod-ca-cert\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.232661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.227695 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-certs\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.298618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.288530 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-podinfo\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.298618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.291178 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jn6\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-kube-api-access-79jn6\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.324285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.311593 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-token\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.324285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.316118 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-envoy\") pod \"e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.552954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.552856 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:45.628994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.628956 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:45.834872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.834787 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:45.915052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.915021 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl"] Nov 01 22:46:45.915689 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:46:45.915649 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49ed163_c875_4004_9f6e_3a5059d8ccf1.slice/crio-d42fb4be0c98bd2584b533a47abf13939087c369b1ea560f894b717cd5cfb839 WatchSource:0}: Error finding container d42fb4be0c98bd2584b533a47abf13939087c369b1ea560f894b717cd5cfb839: Status 404 returned error can't find the container with id d42fb4be0c98bd2584b533a47abf13939087c369b1ea560f894b717cd5cfb839 Nov 01 22:46:45.919359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.919320 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:45.919467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.919411 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:45.919467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:45.919458 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:46.121324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.121236 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:46.422182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.422083 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" event={"ID":"e49ed163-c875-4004-9f6e-3a5059d8ccf1","Type":"ContainerStarted","Data":"8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928"} Nov 01 22:46:46.422514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.422491 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" event={"ID":"e49ed163-c875-4004-9f6e-3a5059d8ccf1","Type":"ContainerStarted","Data":"d42fb4be0c98bd2584b533a47abf13939087c369b1ea560f894b717cd5cfb839"} Nov 01 22:46:46.460726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.460674 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podStartSLOduration=2.460659759 podStartE2EDuration="2.460659759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:46.452117078 +0000 UTC m=+1371.879995010" watchObservedRunningTime="2026-11-01 22:46:46.460659759 +0000 UTC m=+1371.888537689" Nov 01 22:46:46.553838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.553798 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:46.603457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.603410 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:46.743864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.743834 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb"] Nov 01 22:46:46.822803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.822714 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:46.924641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:46.924587 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:47.443791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:47.443754 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:46:47.443976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:47.443813 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:46:47.563522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:47.563486 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:47.690260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:47.690187 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:48.446943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:48.446873 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:48.448170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:48.448142 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:46:48.455462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:48.455434 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:48.932657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:48.932625 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:49.089700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:49.089664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:49.333734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:49.333649 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:49.601109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:49.600995 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:49.793074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:49.793043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:49.796717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:49.796626 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57"] Nov 01 22:46:49.817239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:49.817208 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30578128-bpl57"] Nov 01 22:46:50.462460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:50.462421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:50.676201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:50.666393 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:50.958090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:50.958058 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0964fac4-218a-4183-851f-77b27741786c" path="/var/lib/kubelet/pods/0964fac4-218a-4183-851f-77b27741786c/volumes" Nov 01 22:46:51.064751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:51.064709 4981 patch_prober.go:29] interesting pod/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:46:51.064930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:51.064787 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:46:51.676658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:51.676626 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:52.114417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:51.744715 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" podUID="76eae60f-3b64-434d-8d16-0d9bd4412be6" containerName="payload-processing" containerID="cri-o://b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1" gracePeriod=25 Nov 01 22:46:52.114417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.063305 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:46:52.124665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.124283 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76eae60f-3b64-434d-8d16-0d9bd4412be6-kube-api-access-ld6n2\" (UniqueName: \"kubernetes.io/projected/76eae60f-3b64-434d-8d16-0d9bd4412be6-kube-api-access-ld6n2\") pod \"76eae60f-3b64-434d-8d16-0d9bd4412be6\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " Nov 01 22:46:52.124665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.124394 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76eae60f-3b64-434d-8d16-0d9bd4412be6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76eae60f-3b64-434d-8d16-0d9bd4412be6-plugins-config-volume\") pod \"76eae60f-3b64-434d-8d16-0d9bd4412be6\" (UID: \"76eae60f-3b64-434d-8d16-0d9bd4412be6\") " Nov 01 22:46:52.124931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.124891 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76eae60f-3b64-434d-8d16-0d9bd4412be6-plugins-config-volume" pod "76eae60f-3b64-434d-8d16-0d9bd4412be6" (UID: "76eae60f-3b64-434d-8d16-0d9bd4412be6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:46:52.125494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.125469 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76eae60f-3b64-434d-8d16-0d9bd4412be6-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:52.126646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.126609 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76eae60f-3b64-434d-8d16-0d9bd4412be6-kube-api-access-ld6n2" pod "76eae60f-3b64-434d-8d16-0d9bd4412be6" (UID: "76eae60f-3b64-434d-8d16-0d9bd4412be6"). InnerVolumeSpecName "kube-api-access-ld6n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:46:52.227394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.227365 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld6n2\" (UniqueName: \"kubernetes.io/projected/76eae60f-3b64-434d-8d16-0d9bd4412be6-kube-api-access-ld6n2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:46:52.488628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.488590 4981 generic.go:356] "Generic (PLEG): container finished" podID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerID="386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18" exitCode=0 Nov 01 22:46:52.488822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.488651 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" event={"ID":"6c078984-19f8-4130-9ec5-6ae240e8557d","Type":"ContainerDied","Data":"386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18"} Nov 01 22:46:52.489235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.489216 4981 scope.go:122] "RemoveContainer" containerID="386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18" Nov 01 22:46:52.490134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.490118 4981 generic.go:356] "Generic (PLEG): container finished" podID="76eae60f-3b64-434d-8d16-0d9bd4412be6" containerID="b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1" exitCode=0 Nov 01 22:46:52.490215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.490168 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" Nov 01 22:46:52.490268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.490207 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" event={"ID":"76eae60f-3b64-434d-8d16-0d9bd4412be6","Type":"ContainerDied","Data":"b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1"} Nov 01 22:46:52.490326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.490276 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb" event={"ID":"76eae60f-3b64-434d-8d16-0d9bd4412be6","Type":"ContainerDied","Data":"588ed30b58249e2860c4c0e122cf1c94356f2ca8f836e111d5c023a38c2279b6"} Nov 01 22:46:52.490326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.490308 4981 scope.go:122] "RemoveContainer" containerID="b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1" Nov 01 22:46:52.491478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.491442 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:52.491587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.491535 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:52.491664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.491604 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:52.492211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.492150 4981 generic.go:356] "Generic (PLEG): container finished" podID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerID="e8683695c4b043a8ac74442d9fcc7499a54b39eacfbcd95b65c867a48b674f20" exitCode=0 Nov 01 22:46:52.492266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.492214 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" event={"ID":"5a34390f-1892-407b-b5a6-bc3184b7bc09","Type":"ContainerDied","Data":"e8683695c4b043a8ac74442d9fcc7499a54b39eacfbcd95b65c867a48b674f20"} Nov 01 22:46:52.492706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.492687 4981 scope.go:122] "RemoveContainer" containerID="e8683695c4b043a8ac74442d9fcc7499a54b39eacfbcd95b65c867a48b674f20" Nov 01 22:46:52.494690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.494661 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:52.494763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.494742 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:52.494811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.494790 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:46:52.509736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.509719 4981 scope.go:122] "RemoveContainer" containerID="b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1" Nov 01 22:46:52.510054 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:46:52.510033 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1\": container with ID starting with b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1 not found: ID does not exist" containerID="b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1" Nov 01 22:46:52.510150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.510064 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1"} err="failed to get container status \"b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1\": rpc error: code = NotFound desc = could not find container \"b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1\": container with ID starting with b4c7d7a1feb02237fe1d546ce2e6ed6bdfc94e87c523eace47d49f23335632b1 not found: ID does not exist" Nov 01 22:46:52.606312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.605171 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb"] Nov 01 22:46:52.618201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.618174 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-30578128-5cb56c7555-7gnsb"] Nov 01 22:46:52.758151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.758077 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76eae60f-3b64-434d-8d16-0d9bd4412be6" path="/var/lib/kubelet/pods/76eae60f-3b64-434d-8d16-0d9bd4412be6/volumes" Nov 01 22:46:52.761005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:52.760970 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:53.533786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:53.533726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:53.539196 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:53.539168 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" event={"ID":"5a34390f-1892-407b-b5a6-bc3184b7bc09","Type":"ContainerStarted","Data":"32f85cd6c3dcc03eaf0e29ba0bdbf57b2f7dd2e23fb333355d6dac5216f1207c"} Nov 01 22:46:53.541499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:53.541475 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" event={"ID":"6c078984-19f8-4130-9ec5-6ae240e8557d","Type":"ContainerStarted","Data":"26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4"} Nov 01 22:46:53.610594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:53.610107 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:46:53.610594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:53.610253 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:46:53.610594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:53.610292 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 22:46:54.222701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.211992 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:46:54.222701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.212121 4981 patch_prober.go:29] interesting pod/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:46:54.222701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.212164 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 22:46:54.309601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.298136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:54.600655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.599608 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:46:54.600655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.599674 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 22:46:54.641388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.636050 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7"] Nov 01 22:46:54.641388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:54.636290 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" containerID="cri-o://0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9" gracePeriod=30 Nov 01 22:46:55.062243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.062212 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb"] Nov 01 22:46:55.062648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.062620 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" containerID="cri-o://629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6" gracePeriod=30 Nov 01 22:46:55.227676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.227205 4981 patch_prober.go:29] interesting pod/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:46:55.227676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.227265 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 22:46:55.278599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.267242 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:55.401699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.398036 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:55.616918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.614692 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:46:55.616918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:55.614755 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 22:46:56.015192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:56.015146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:56.211599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:56.210894 4981 patch_prober.go:29] interesting pod/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:46:56.211599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:56.210966 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 22:46:56.628603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:56.620622 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:46:56.628603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:56.620681 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 22:46:57.031075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:57.030499 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:57.171201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:57.170505 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:57.248426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:57.223717 4981 patch_prober.go:29] interesting pod/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:46:57.248426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:57.223784 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 22:46:57.266653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:57.265836 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:57.633630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:57.610595 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:46:57.634464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:57.634426 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 22:46:58.208003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:58.196518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:58.211715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:58.211683 4981 patch_prober.go:29] interesting pod/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:46:58.211978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:58.211944 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 22:46:58.595086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:58.594993 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 01 22:46:58.595359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:58.595325 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 01 22:46:58.998510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:58.998470 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:59.209703 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.209648 4981 patch_prober.go:29] interesting pod/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 01 22:46:59.210045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.209997 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 01 22:46:59.297021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.296910 4981 scope.go:122] "RemoveContainer" containerID="277fa3098e5fdec4fb6cd51c295594f18d3de87cc11f84173ce34029baa6dee3" Nov 01 22:46:59.312379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.312351 4981 scope.go:122] "RemoveContainer" containerID="617d8f1f36fd1bfefc149432f0e3f9787453d6d62235458420a6a21dd7da262d" Nov 01 22:46:59.323761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.323737 4981 scope.go:122] "RemoveContainer" containerID="38f8dd33671d33be8007773adb87966853ae889c833aed4a4cf876d6579332ff" Nov 01 22:46:59.335087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.335065 4981 scope.go:122] "RemoveContainer" containerID="211a28546bd36d8752c5fa2e4a0e7406dce28e0075166f29e698c670c4f009c9" Nov 01 22:46:59.348710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.348686 4981 scope.go:122] "RemoveContainer" containerID="5e42c57b3be3bb3b46adb2354057517e98c01e129d5834258345248e6f201db5" Nov 01 22:46:59.367840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.367813 4981 scope.go:122] "RemoveContainer" containerID="402042e1d0c3173d8cb1d7e1d84db4da3c7e16f5e028eceb98f31681d63ec140" Nov 01 22:46:59.384115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.384079 4981 scope.go:122] "RemoveContainer" containerID="bc5d5541b0f3af08b9c26f5c90ad35daaa04e6a4de9495c4b1209412466af019" Nov 01 22:46:59.398570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.398530 4981 scope.go:122] "RemoveContainer" containerID="a823c75c01a7405288e9d77268b31f84db96dc0bac3d90a22b9bc58b8dc1ef08" Nov 01 22:46:59.410126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.410108 4981 scope.go:122] "RemoveContainer" containerID="12d3c4a2db9e8e490effae70be6490c2581ee7fcae488b9296f0f5441dd24d30" Nov 01 22:46:59.420832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.420807 4981 scope.go:122] "RemoveContainer" containerID="b695e6dc275e64d50b9f0d99102ed791cfa0c1ef45387352360a2bef3e08527c" Nov 01 22:46:59.438890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.434666 4981 scope.go:122] "RemoveContainer" containerID="ae8557051f27b94b73af6b87311a0bec0ce0f4b2b58d793235885b691c77eb4d" Nov 01 22:46:59.463772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.463742 4981 scope.go:122] "RemoveContainer" containerID="2d904167066cb89d67a11d063d27b4382b421e814871652eed6d10f2610472e0" Nov 01 22:46:59.481185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.481160 4981 scope.go:122] "RemoveContainer" containerID="2fa7b5d3c9f255115424b50bd23e77a7f6e97e4556499d69e0cee9a71935e10a" Nov 01 22:46:59.496293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.496269 4981 scope.go:122] "RemoveContainer" containerID="8eea1f306afd3627fc4c06c6f957a0fe363e2828b6a37b3b6f065fb71e5531ab" Nov 01 22:46:59.510839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.510806 4981 scope.go:122] "RemoveContainer" containerID="b26a48f87d171e62c1ce8c9de92eea214b135575367f3349d4e6ea169954b049" Nov 01 22:46:59.526677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.526615 4981 scope.go:122] "RemoveContainer" containerID="29028ff8399b45af117f22fdc5c5148c142891f9e991dfd658c0e191360b62d7" Nov 01 22:46:59.611079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.610897 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:46:59.611650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.611628 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:46:59.612348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.612330 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 22:46:59.878050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.878000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:46:59.981265 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:46:59.981227 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:47:00.100929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.100898 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:00.111065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111040 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-data\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111084 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-socket\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111143 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d42e2f-571f-4e4f-9913-f1510b869444-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d42e2f-571f-4e4f-9913-f1510b869444-istiod-ca-cert\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111196 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-kube-api-access-csxch\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-kube-api-access-csxch\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111241 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-credential-socket\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111343 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-data" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.111384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111359 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-socket" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.111537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111443 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-istio-token\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-istio-token\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111590 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-credential-socket" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.111675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111630 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-certs\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111636 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d42e2f-571f-4e4f-9913-f1510b869444-istiod-ca-cert" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:00.111752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111691 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04d42e2f-571f-4e4f-9913-f1510b869444-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d42e2f-571f-4e4f-9913-f1510b869444-istio-podinfo\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.111824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111806 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-certs" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.111880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.111865 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-envoy\") pod \"04d42e2f-571f-4e4f-9913-f1510b869444\" (UID: \"04d42e2f-571f-4e4f-9913-f1510b869444\") " Nov 01 22:47:00.112754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.112731 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.112754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.112752 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.112939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.112761 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.112939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.112772 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04d42e2f-571f-4e4f-9913-f1510b869444-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.112939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.112781 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.113754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.113730 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-istio-token" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.113874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.113760 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-kube-api-access-csxch" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "kube-api-access-csxch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.114114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.114100 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-envoy" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.114157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.114111 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04d42e2f-571f-4e4f-9913-f1510b869444-istio-podinfo" pod "04d42e2f-571f-4e4f-9913-f1510b869444" (UID: "04d42e2f-571f-4e4f-9913-f1510b869444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:00.216073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.215963 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04d42e2f-571f-4e4f-9913-f1510b869444-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.216073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.215999 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04d42e2f-571f-4e4f-9913-f1510b869444-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.216073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.216014 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csxch\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-kube-api-access-csxch\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.216073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.216029 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04d42e2f-571f-4e4f-9913-f1510b869444-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.219608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.216906 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:47:00.219608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.217824 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:47:00.219608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.219179 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:47:00.375054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.375013 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:47:00.422562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.422502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.528561 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istiod-ca-cert\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.528728 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-credential-socket\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.528816 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-socket\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.528925 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-envoy\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.528955 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-podinfo\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.529308 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-credential-socket" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.529523 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-data\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.529613 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-token\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-token\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.529697 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-socket" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.529717 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-certs\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.529792 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-kube-api-access-b7zv8\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-kube-api-access-b7zv8\") pod \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\" (UID: \"2a82c996-36f9-4a94-803b-f78f1eaa8bdd\") " Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.530100 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-data" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.530315 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-certs" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.530762 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istiod-ca-cert" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.531020 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.531043 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.531061 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.531075 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.531537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.531089 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.533157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.533135 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-kube-api-access-b7zv8" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "kube-api-access-b7zv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.533287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.533268 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-envoy" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:00.533335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.533290 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-token" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:00.533495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.533482 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-podinfo" pod "2a82c996-36f9-4a94-803b-f78f1eaa8bdd" (UID: "2a82c996-36f9-4a94-803b-f78f1eaa8bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:00.632418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.632336 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.632418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.632364 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zv8\" (UniqueName: \"kubernetes.io/projected/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-kube-api-access-b7zv8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.632418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.632374 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.632418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.632382 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a82c996-36f9-4a94-803b-f78f1eaa8bdd-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:00.737845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.737808 4981 generic.go:356] "Generic (PLEG): container finished" podID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerID="629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6" exitCode=0 Nov 01 22:47:00.738038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.737937 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" Nov 01 22:47:00.744222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.744181 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" event={"ID":"2a82c996-36f9-4a94-803b-f78f1eaa8bdd","Type":"ContainerDied","Data":"629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6"} Nov 01 22:47:00.744405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.744251 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb" event={"ID":"2a82c996-36f9-4a94-803b-f78f1eaa8bdd","Type":"ContainerDied","Data":"afaa2cdff6920c15efef0976bcd2d5ac29a2a56542949ceabf9e170457e44ea6"} Nov 01 22:47:00.744405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.744285 4981 scope.go:122] "RemoveContainer" containerID="629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6" Nov 01 22:47:00.747653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.747626 4981 generic.go:356] "Generic (PLEG): container finished" podID="04d42e2f-571f-4e4f-9913-f1510b869444" containerID="0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9" exitCode=0 Nov 01 22:47:00.748323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.748301 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" Nov 01 22:47:00.749326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.749298 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" event={"ID":"04d42e2f-571f-4e4f-9913-f1510b869444","Type":"ContainerDied","Data":"0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9"} Nov 01 22:47:00.749437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.749347 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7" event={"ID":"04d42e2f-571f-4e4f-9913-f1510b869444","Type":"ContainerDied","Data":"ae0b92259bf917d1a2b82302dd46c34d284189a49f7c103cb5945776d213d8a8"} Nov 01 22:47:00.766921 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.766840 4981 scope.go:122] "RemoveContainer" containerID="629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6" Nov 01 22:47:00.767708 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:47:00.767675 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6\": container with ID starting with 629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6 not found: ID does not exist" containerID="629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6" Nov 01 22:47:00.767843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.767717 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6"} err="failed to get container status \"629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6\": rpc error: code = NotFound desc = could not find container \"629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6\": container with ID starting with 629ef218e687be2512b07e53fa7bb9c5fcacf3e702083d70154607c7358d44f6 not found: ID does not exist" Nov 01 22:47:00.767843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.767745 4981 scope.go:122] "RemoveContainer" containerID="0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9" Nov 01 22:47:00.769007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.768910 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2"] Nov 01 22:47:00.770214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770191 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" Nov 01 22:47:00.770214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770213 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" Nov 01 22:47:00.770386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770244 4981 cpu_manager.go:414] "containerMap: removing container" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" Nov 01 22:47:00.770386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770252 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" Nov 01 22:47:00.770386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770264 4981 cpu_manager.go:414] "containerMap: removing container" podUID="76eae60f-3b64-434d-8d16-0d9bd4412be6" containerName="payload-processing" Nov 01 22:47:00.770386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770272 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76eae60f-3b64-434d-8d16-0d9bd4412be6" containerName="payload-processing" Nov 01 22:47:00.770715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770486 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" containerName="istio-proxy" Nov 01 22:47:00.770715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770508 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" containerName="istio-proxy" Nov 01 22:47:00.770715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.770519 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="76eae60f-3b64-434d-8d16-0d9bd4412be6" containerName="payload-processing" Nov 01 22:47:00.783447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.783420 4981 scope.go:122] "RemoveContainer" containerID="0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9" Nov 01 22:47:00.786521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.785613 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.786769 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:47:00.786741 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9\": container with ID starting with 0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9 not found: ID does not exist" containerID="0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9" Nov 01 22:47:00.790856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.787338 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9"} err="failed to get container status \"0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9\": rpc error: code = NotFound desc = could not find container \"0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9\": container with ID starting with 0b8e037edd2e51eccfc60594ce8273cfb93d8b89ca98c50dcc49484d6df67de9 not found: ID does not exist" Nov 01 22:47:00.791732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.791710 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-44a9aa-openshift-default-dockercfg-h6v7m\"" Nov 01 22:47:00.804536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.804064 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2"] Nov 01 22:47:00.817998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.813182 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb"] Nov 01 22:47:00.823099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.823062 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-kbdrb"] Nov 01 22:47:00.831029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.830990 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.940806 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.940874 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.940960 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8kjx\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-kube-api-access-b8kjx\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.941004 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istiod-ca-cert\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.941169 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-credential-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.941259 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.941293 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-data\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.941331 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-certs\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:00.965604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:00.941359 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-podinfo\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.023615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.011885 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:01.063606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.048954 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.165545 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.165696 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8kjx\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-kube-api-access-b8kjx\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.165745 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istiod-ca-cert\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.166096 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-credential-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.166217 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.166252 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-data\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.166293 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-certs\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.167658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.166320 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-podinfo\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.176413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.175036 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istiod-ca-cert\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.177051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.062371 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.177775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.160654 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7"] Nov 01 22:47:01.178169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.178137 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-data\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.178547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.178522 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-certs\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.178932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.178902 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-credential-socket\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.185347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.185314 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-llbl7"] Nov 01 22:47:01.187446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.187399 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.190338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.190308 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8kjx\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-kube-api-access-b8kjx\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.190736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.190480 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-podinfo\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.192274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.192253 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\") pod \"e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.429422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.429390 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:01.460791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.460756 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:01.735545 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:47:01.735510 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1489db7_ad64_4845_8fdc_2474a55aaf0e.slice/crio-7801be417e86792e6aa272cafa34dee2903dcb71ac1f02ec36b4a8520573a649 WatchSource:0}: Error finding container 7801be417e86792e6aa272cafa34dee2903dcb71ac1f02ec36b4a8520573a649: Status 404 returned error can't find the container with id 7801be417e86792e6aa272cafa34dee2903dcb71ac1f02ec36b4a8520573a649 Nov 01 22:47:01.736957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.736830 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2"] Nov 01 22:47:01.739367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.739322 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:01.739452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.739411 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:01.739531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.739457 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:01.764257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:01.764223 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" event={"ID":"b1489db7-ad64-4845-8fdc-2474a55aaf0e","Type":"ContainerStarted","Data":"7801be417e86792e6aa272cafa34dee2903dcb71ac1f02ec36b4a8520573a649"} Nov 01 22:47:02.540099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:02.540065 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:02.738777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:02.738743 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d42e2f-571f-4e4f-9913-f1510b869444" path="/var/lib/kubelet/pods/04d42e2f-571f-4e4f-9913-f1510b869444/volumes" Nov 01 22:47:02.739327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:02.739312 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a82c996-36f9-4a94-803b-f78f1eaa8bdd" path="/var/lib/kubelet/pods/2a82c996-36f9-4a94-803b-f78f1eaa8bdd/volumes" Nov 01 22:47:02.776768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:02.776730 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" event={"ID":"b1489db7-ad64-4845-8fdc-2474a55aaf0e","Type":"ContainerStarted","Data":"cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463"} Nov 01 22:47:02.819802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:02.819667 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podStartSLOduration=2.81965101 podStartE2EDuration="2.81965101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:02.816416966 +0000 UTC m=+1388.244294898" watchObservedRunningTime="2026-11-01 22:47:02.81965101 +0000 UTC m=+1388.247528940" Nov 01 22:47:03.429630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:03.429592 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:03.435089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:03.435059 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:03.566680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:03.566634 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:03.793058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:03.792937 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:03.794264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:03.794238 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:47:04.721396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:04.721357 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:05.017897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:05.017799 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:05.139991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:05.139958 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:05.743075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:05.743039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:06.444543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.444510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:06.507487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.507454 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj"] Nov 01 22:47:06.514077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.514054 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:06.526458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.526434 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj"] Nov 01 22:47:06.531473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.531451 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs"] Nov 01 22:47:06.533742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.533717 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-44a9aa\"" Nov 01 22:47:06.533880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.533745 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-44a9aa-dockercfg-kgmxn\"" Nov 01 22:47:06.537541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.537520 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:06.557690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.557661 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs"] Nov 01 22:47:06.581473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.581443 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:06.692602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.691756 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hd5r\" (UniqueName: \"kubernetes.io/projected/ca59b420-9ae9-4196-9cb4-c01f68908a50-kube-api-access-9hd5r\") pod \"payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:06.692602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.692142 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjql\" (UniqueName: \"kubernetes.io/projected/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-kube-api-access-dwjql\") pod \"payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:06.692602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.692222 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:06.692602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.692308 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca59b420-9ae9-4196-9cb4-c01f68908a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:06.826676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.820611 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjql\" (UniqueName: \"kubernetes.io/projected/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-kube-api-access-dwjql\") pod \"payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:06.826676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.820720 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:06.826676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.820810 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca59b420-9ae9-4196-9cb4-c01f68908a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:06.826676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.820852 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hd5r\" (UniqueName: \"kubernetes.io/projected/ca59b420-9ae9-4196-9cb4-c01f68908a50-kube-api-access-9hd5r\") pod \"payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:06.859188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.852359 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca59b420-9ae9-4196-9cb4-c01f68908a50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:06.859188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.852450 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:06.887194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.887114 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hd5r\" (UniqueName: \"kubernetes.io/projected/ca59b420-9ae9-4196-9cb4-c01f68908a50-kube-api-access-9hd5r\") pod \"payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:06.920325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.920256 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjql\" (UniqueName: \"kubernetes.io/projected/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-kube-api-access-dwjql\") pod \"payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:06.993108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:06.993070 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:07.103332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.103228 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:07.131468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.131434 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:07.160622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.160554 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:07.226734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.224468 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:07.354335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.354244 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:07.401472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.401438 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8"] Nov 01 22:47:07.411685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.407853 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.416119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.416090 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e202e7-openshift-default-dockercfg-s5jpf\"" Nov 01 22:47:07.435056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.435025 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8"] Nov 01 22:47:07.466322 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:47:07.466292 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca59b420_9ae9_4196_9cb4_c01f68908a50.slice/crio-09545c5cb0d6767a0ce143d1b317b78182b571a797fbae71bd132a0c448c16b2 WatchSource:0}: Error finding container 09545c5cb0d6767a0ce143d1b317b78182b571a797fbae71bd132a0c448c16b2: Status 404 returned error can't find the container with id 09545c5cb0d6767a0ce143d1b317b78182b571a797fbae71bd132a0c448c16b2 Nov 01 22:47:07.468502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.468460 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj"] Nov 01 22:47:07.495807 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:47:07.495768 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0ae1bb_f3ae_4f70_b9a0_1b0832cea7a2.slice/crio-26b56b33d38576175684af1f8f8747e58bbfe2337891f37b89985931f73c5ad9 WatchSource:0}: Error finding container 26b56b33d38576175684af1f8f8747e58bbfe2337891f37b89985931f73c5ad9: Status 404 returned error can't find the container with id 26b56b33d38576175684af1f8f8747e58bbfe2337891f37b89985931f73c5ad9 Nov 01 22:47:07.498288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.497196 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs"] Nov 01 22:47:07.502450 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.502409 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.583745 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996lr\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-kube-api-access-996lr\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.583849 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.583943 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.583972 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.584036 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.584161 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.584226 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.584275 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.604680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.584355 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.635322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.635156 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.689726 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.689826 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.689914 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.690016 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-996lr\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-kube-api-access-996lr\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.690111 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.690200 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.690230 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.690305 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.690443 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.690907 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-certs\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.691599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.691441 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istiod-ca-cert\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.692686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.692664 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-credential-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.705763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.702804 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-data\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.705763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.703000 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-socket\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.715055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.710446 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-envoy\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.741441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.741372 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-podinfo\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.753609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.745383 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-token\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.754973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.754907 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-996lr\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-kube-api-access-996lr\") pod \"e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:07.982198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.982153 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" event={"ID":"ca59b420-9ae9-4196-9cb4-c01f68908a50","Type":"ContainerStarted","Data":"a950c8fb96b08e416668008ecac7dc565708bf0ca5957bafddf0e8af91533e4d"} Nov 01 22:47:07.982704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.982226 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:07.982704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.982254 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" event={"ID":"ca59b420-9ae9-4196-9cb4-c01f68908a50","Type":"ContainerStarted","Data":"09545c5cb0d6767a0ce143d1b317b78182b571a797fbae71bd132a0c448c16b2"} Nov 01 22:47:07.984169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.984134 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" event={"ID":"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2","Type":"ContainerStarted","Data":"c0361a14518e2227e9c8ff497a12bcedd1582a9677d4197c96e506963fbd3ca8"} Nov 01 22:47:07.984278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.984183 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" event={"ID":"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2","Type":"ContainerStarted","Data":"26b56b33d38576175684af1f8f8747e58bbfe2337891f37b89985931f73c5ad9"} Nov 01 22:47:07.984278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:07.984235 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:08.006226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.006180 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" podStartSLOduration=2.006166539 podStartE2EDuration="2.006166539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:08.005003894 +0000 UTC m=+1393.432881826" watchObservedRunningTime="2026-11-01 22:47:08.006166539 +0000 UTC m=+1393.434044470" Nov 01 22:47:08.031298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.031090 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" podStartSLOduration=2.031070617 podStartE2EDuration="2.031070617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:08.027517909 +0000 UTC m=+1393.455395855" watchObservedRunningTime="2026-11-01 22:47:08.031070617 +0000 UTC m=+1393.458948551" Nov 01 22:47:08.033209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.032954 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:08.095551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.095518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:08.221491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.221459 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:08.225947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.222590 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6"] Nov 01 22:47:08.229627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.229593 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6"] Nov 01 22:47:08.229843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.229737 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.237077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.236998 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:47:08.237374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.237342 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:47:08.237552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.237539 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:47:08.237901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.237879 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:47:08.324372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.324325 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.324643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.324466 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.325662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.325607 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkkk\" (UniqueName: \"kubernetes.io/projected/3eef9e5b-a10a-4bac-b2fa-0d483f069615-kube-api-access-5vkkk\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.356611 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:47:08.356564 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb153d5e5_f84f_48a3_baea_50a3fbce1b12.slice/crio-89581285b6b3b69b12ce0355684a1a31a0bca5770d9acc9f57c35b479c586411 WatchSource:0}: Error finding container 89581285b6b3b69b12ce0355684a1a31a0bca5770d9acc9f57c35b479c586411: Status 404 returned error can't find the container with id 89581285b6b3b69b12ce0355684a1a31a0bca5770d9acc9f57c35b479c586411 Nov 01 22:47:08.356926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.356905 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8"] Nov 01 22:47:08.359588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.359538 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:08.359691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.359623 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:08.359691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.359655 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:08.427268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.427238 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.427411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.427283 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.427411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.427392 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkkk\" (UniqueName: \"kubernetes.io/projected/3eef9e5b-a10a-4bac-b2fa-0d483f069615-kube-api-access-5vkkk\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.428005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.427979 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.429989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.429966 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.436321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.436294 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkkk\" (UniqueName: \"kubernetes.io/projected/3eef9e5b-a10a-4bac-b2fa-0d483f069615-kube-api-access-5vkkk\") pod \"maas-api-revoke-keys-e2e-api-a-2476f8-px2l6\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.553023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.552951 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:08.957601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:08.957547 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6"] Nov 01 22:47:09.022275 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:47:09.017527 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eef9e5b_a10a_4bac_b2fa_0d483f069615.slice/crio-ef874641d693a25f72ced2a022b5b4aaddade72e5e5378641ce09d44f2d21729 WatchSource:0}: Error finding container ef874641d693a25f72ced2a022b5b4aaddade72e5e5378641ce09d44f2d21729: Status 404 returned error can't find the container with id ef874641d693a25f72ced2a022b5b4aaddade72e5e5378641ce09d44f2d21729 Nov 01 22:47:09.036497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:09.034552 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" event={"ID":"b153d5e5-f84f-48a3-baea-50a3fbce1b12","Type":"ContainerStarted","Data":"3114dd87e4bdcffc34b982431cb285ff3f4c58026bb0a642aea0ac0255ca50a9"} Nov 01 22:47:09.036497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:09.034655 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" event={"ID":"b153d5e5-f84f-48a3-baea-50a3fbce1b12","Type":"ContainerStarted","Data":"89581285b6b3b69b12ce0355684a1a31a0bca5770d9acc9f57c35b479c586411"} Nov 01 22:47:09.094192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:09.094129 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podStartSLOduration=2.094095168 podStartE2EDuration="2.094095168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:09.052267885 +0000 UTC m=+1394.480145813" watchObservedRunningTime="2026-11-01 22:47:09.094095168 +0000 UTC m=+1394.521973104" Nov 01 22:47:09.566917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:09.566834 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:10.033325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.033294 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:10.039829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.039805 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:10.045392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.045352 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" event={"ID":"3eef9e5b-a10a-4bac-b2fa-0d483f069615","Type":"ContainerStarted","Data":"9984df973612e97eb62d45da6ad821d2bf34ebfdc7de32b8539c0860980adb6c"} Nov 01 22:47:10.045509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.045439 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:10.045509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.045470 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" event={"ID":"3eef9e5b-a10a-4bac-b2fa-0d483f069615","Type":"ContainerStarted","Data":"ef874641d693a25f72ced2a022b5b4aaddade72e5e5378641ce09d44f2d21729"} Nov 01 22:47:10.046268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.046247 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:47:10.124826 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.124777 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" podStartSLOduration=2.124763214 podStartE2EDuration="2.124763214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:10.121011012 +0000 UTC m=+1395.548888944" watchObservedRunningTime="2026-11-01 22:47:10.124763214 +0000 UTC m=+1395.552641144" Nov 01 22:47:10.679197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:10.679168 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:11.056039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:11.055954 4981 generic.go:356] "Generic (PLEG): container finished" podID="3eef9e5b-a10a-4bac-b2fa-0d483f069615" containerID="9984df973612e97eb62d45da6ad821d2bf34ebfdc7de32b8539c0860980adb6c" exitCode=0 Nov 01 22:47:11.056411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:11.056046 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" event={"ID":"3eef9e5b-a10a-4bac-b2fa-0d483f069615","Type":"ContainerDied","Data":"9984df973612e97eb62d45da6ad821d2bf34ebfdc7de32b8539c0860980adb6c"} Nov 01 22:47:11.380868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:11.380788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:11.509850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:11.509816 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:11.739142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:11.739113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:12.243571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.243548 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:12.393952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.393865 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eef9e5b-a10a-4bac-b2fa-0d483f069615-kube-api-access-5vkkk\" (UniqueName: \"kubernetes.io/projected/3eef9e5b-a10a-4bac-b2fa-0d483f069615-kube-api-access-5vkkk\") pod \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " Nov 01 22:47:12.394117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.394025 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-openshift-ca\") pod \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " Nov 01 22:47:12.394221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.394174 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-xks-ca\") pod \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\" (UID: \"3eef9e5b-a10a-4bac-b2fa-0d483f069615\") " Nov 01 22:47:12.394406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.394377 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-openshift-ca" pod "3eef9e5b-a10a-4bac-b2fa-0d483f069615" (UID: "3eef9e5b-a10a-4bac-b2fa-0d483f069615"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:12.395028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.395005 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:12.396160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.396139 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eef9e5b-a10a-4bac-b2fa-0d483f069615-kube-api-access-5vkkk" pod "3eef9e5b-a10a-4bac-b2fa-0d483f069615" (UID: "3eef9e5b-a10a-4bac-b2fa-0d483f069615"). InnerVolumeSpecName "kube-api-access-5vkkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:12.396297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.396282 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-xks-ca" pod "3eef9e5b-a10a-4bac-b2fa-0d483f069615" (UID: "3eef9e5b-a10a-4bac-b2fa-0d483f069615"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:47:12.496442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.496410 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vkkk\" (UniqueName: \"kubernetes.io/projected/3eef9e5b-a10a-4bac-b2fa-0d483f069615-kube-api-access-5vkkk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:12.496442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.496441 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3eef9e5b-a10a-4bac-b2fa-0d483f069615-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:12.616288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.616251 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:12.740271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:12.740232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:13.144630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:13.144509 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" Nov 01 22:47:13.144964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:13.144536 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6" event={"ID":"3eef9e5b-a10a-4bac-b2fa-0d483f069615","Type":"ContainerDied","Data":"ef874641d693a25f72ced2a022b5b4aaddade72e5e5378641ce09d44f2d21729"} Nov 01 22:47:13.145049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:13.144993 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef874641d693a25f72ced2a022b5b4aaddade72e5e5378641ce09d44f2d21729" Nov 01 22:47:13.623933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:13.623902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:13.731322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:13.731294 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:14.526218 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:47:14.526186 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:47:14.691920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:14.691889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:15.481475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:15.481445 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:15.743532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:15.743455 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:17.100828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:17.100797 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:17.382760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:17.382659 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:18.091813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:18.091777 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:18.410143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:18.410012 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:18.503445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:18.503408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:19.052374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:19.052332 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:47:19.054732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:19.054708 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:47:19.372935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:19.372825 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:20.478769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:20.478734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:21.604762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:21.604731 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:23.030770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:23.030731 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:23.770551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:23.770502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:24.370603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:24.368796 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:24.834597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:24.833678 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:25.699365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:25.692626 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:25.842551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:25.842513 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:26.320643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:26.320560 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:27.530332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:27.530290 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:27.637763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:27.637725 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:27.793395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:27.793247 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:28.942014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:28.941980 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:29.305073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:29.305038 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:29.457001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:29.456969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:30.033772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.033737 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f"] Nov 01 22:47:30.040028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.039994 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bda0c090-mtm8f"] Nov 01 22:47:30.040657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.040627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:30.543513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.543478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:30.643790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.643756 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc"] Nov 01 22:47:30.644668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.644619 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3eef9e5b-a10a-4bac-b2fa-0d483f069615" containerName="revoke-keys" Nov 01 22:47:30.644668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.644636 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eef9e5b-a10a-4bac-b2fa-0d483f069615" containerName="revoke-keys" Nov 01 22:47:30.644797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.644765 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eef9e5b-a10a-4bac-b2fa-0d483f069615" containerName="revoke-keys" Nov 01 22:47:30.650045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.650023 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:30.675958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.675909 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-58c957\"" Nov 01 22:47:30.680092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.676203 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-58c957-dockercfg-h9297\"" Nov 01 22:47:30.694241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.694161 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc"] Nov 01 22:47:30.695995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.695626 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a1edad-7830-497a-9de1-bc320ed49297-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:30.695995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.695826 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/a1a1edad-7830-497a-9de1-bc320ed49297-kube-api-access-4vlvg\") pod \"payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:30.822657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.813507 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a1edad-7830-497a-9de1-bc320ed49297-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:30.859122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.844630 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a1edad-7830-497a-9de1-bc320ed49297-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:30.860392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.860360 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/a1a1edad-7830-497a-9de1-bc320ed49297-kube-api-access-4vlvg\") pod \"payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:30.861748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.861641 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:30.900640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:30.895074 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/a1a1edad-7830-497a-9de1-bc320ed49297-kube-api-access-4vlvg\") pod \"payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:31.011167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:31.011125 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:31.064686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:31.044207 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434a2293-8f5a-4e20-8e20-7d3ca598ed8d" path="/var/lib/kubelet/pods/434a2293-8f5a-4e20-8e20-7d3ca598ed8d/volumes" Nov 01 22:47:31.371651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:31.371557 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:31.399039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:31.399007 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc"] Nov 01 22:47:32.074963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:32.074926 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" event={"ID":"a1a1edad-7830-497a-9de1-bc320ed49297","Type":"ContainerStarted","Data":"4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4"} Nov 01 22:47:32.075488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:32.074992 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" event={"ID":"a1a1edad-7830-497a-9de1-bc320ed49297","Type":"ContainerStarted","Data":"f20431be9ac899d7fe988236d512b8b0fb15f35c267d85af8d1e3ec856a42530"} Nov 01 22:47:32.075488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:32.075174 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:32.126157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:32.126104 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" podStartSLOduration=2.12609156 podStartE2EDuration="2.12609156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:32.121703319 +0000 UTC m=+1417.549581252" watchObservedRunningTime="2026-11-01 22:47:32.12609156 +0000 UTC m=+1417.553969491" Nov 01 22:47:32.589600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:32.578683 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:32.839160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:32.839108 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:33.869597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:33.869102 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:34.242750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:34.242706 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:47:34.243157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:34.242774 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:47:34.533864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:34.533765 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:34.741260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:34.741230 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:34.860538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:34.860497 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:34.965342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:34.965268 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:35.069649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.066686 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:35.114928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.114895 4981 generic.go:356] "Generic (PLEG): container finished" podID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerID="cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824" exitCode=0 Nov 01 22:47:35.115139 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.114991 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" event={"ID":"8f2dd39f-6f0f-4481-b36d-6d059a93958c","Type":"ContainerDied","Data":"cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824"} Nov 01 22:47:35.115839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.115809 4981 scope.go:122] "RemoveContainer" containerID="cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824" Nov 01 22:47:35.118048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.118004 4981 generic.go:356] "Generic (PLEG): container finished" podID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerID="fb12f37016e03f205114c0f4b7aeed2f215d69516fae46772910ca9b68562acc" exitCode=0 Nov 01 22:47:35.118184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.118080 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" event={"ID":"bcddcb52-6b43-4652-8057-9f898efb0a41","Type":"ContainerDied","Data":"fb12f37016e03f205114c0f4b7aeed2f215d69516fae46772910ca9b68562acc"} Nov 01 22:47:35.118402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.118373 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:35.118489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.118451 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:35.118543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.118490 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:35.118654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.118535 4981 scope.go:122] "RemoveContainer" containerID="fb12f37016e03f205114c0f4b7aeed2f215d69516fae46772910ca9b68562acc" Nov 01 22:47:35.121016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.120985 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:35.121213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.121194 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:35.121356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.121331 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:35.183537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.183490 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:35.231151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.231073 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:47:35.570418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.570380 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft"] Nov 01 22:47:35.605671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.605638 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:35.611918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.611896 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft"] Nov 01 22:47:35.612454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.612431 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.645783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.645758 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e36490-openshift-default-dockercfg-vjrhb\"" Nov 01 22:47:35.801737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.795783 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efbf218-9374-41a1-bcc3-00b296a7b05d-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.801737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.795876 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.801737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.795941 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.801737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.796052 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.822653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.820321 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g778f\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-kube-api-access-g778f\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.822653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.820528 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.822653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.820558 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.822653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.820611 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.822653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.820703 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.845598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.845108 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.929966 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efbf218-9374-41a1-bcc3-00b296a7b05d-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930062 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930124 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930253 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930284 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g778f\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-kube-api-access-g778f\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930422 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930448 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930478 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.930554 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.931170 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.932360 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efbf218-9374-41a1-bcc3-00b296a7b05d-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.932615 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.933372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.932847 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.960602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.951374 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.967607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.961826 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.976598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.971127 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.977607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.976848 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g778f\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-kube-api-access-g778f\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:35.987771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:35.982108 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:36.146047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.145963 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" event={"ID":"8f2dd39f-6f0f-4481-b36d-6d059a93958c","Type":"ContainerStarted","Data":"69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a"} Nov 01 22:47:36.149758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.149725 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" event={"ID":"bcddcb52-6b43-4652-8057-9f898efb0a41","Type":"ContainerStarted","Data":"275819ea563b9e380c0216b96bca273602e0c875cf547863a4f19a91f6b5f583"} Nov 01 22:47:36.253433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.253396 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:36.498816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.497973 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:47:36.534954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.534068 4981 patch_prober.go:29] interesting pod/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:47:36.534954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.534142 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 22:47:36.576041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.576004 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:36.800599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.781463 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft"] Nov 01 22:47:36.833109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.829126 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:36.833109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.829214 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:36.833109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.829244 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:47:36.917600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.914066 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:47:36.923599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.918799 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:47:36.923599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:36.922857 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 01 22:47:37.162140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.162055 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" event={"ID":"9efbf218-9374-41a1-bcc3-00b296a7b05d","Type":"ContainerStarted","Data":"3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7"} Nov 01 22:47:37.162140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.162121 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" event={"ID":"9efbf218-9374-41a1-bcc3-00b296a7b05d","Type":"ContainerStarted","Data":"cc761db51f8b1caa903cdde6eee8b9fbbcc4dd8a94214991aa6ee68ad95131a9"} Nov 01 22:47:37.211601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.211399 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podStartSLOduration=2.21137926 podStartE2EDuration="2.21137926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:37.184429411 +0000 UTC m=+1422.612307344" watchObservedRunningTime="2026-11-01 22:47:37.21137926 +0000 UTC m=+1422.639257193" Nov 01 22:47:37.254251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.254214 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:37.255985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.255956 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:47:37.256119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.256003 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:47:37.480224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.479061 4981 patch_prober.go:29] interesting pod/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:47:37.480224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.479310 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 22:47:37.898653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.898492 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:47:37.898653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.898546 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 01 22:47:37.957396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:37.957351 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:38.258692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:38.258660 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:38.485603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:38.483035 4981 patch_prober.go:29] interesting pod/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:47:38.485603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:38.483104 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 22:47:38.906601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:38.899070 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:47:38.906601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:38.899136 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 01 22:47:38.933141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:38.933102 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:39.225931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.225903 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:39.228782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.228755 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:47:39.482039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.481951 4981 patch_prober.go:29] interesting pod/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:47:39.518548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.518509 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 22:47:39.519181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.507324 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:39.653526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.653499 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:39.909181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.909085 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:47:39.909455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:39.909412 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 01 22:47:40.222614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.217452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:40.481654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.478062 4981 patch_prober.go:29] interesting pod/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:47:40.481654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.478133 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 22:47:40.915663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.915003 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:47:40.915663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.915068 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 01 22:47:40.931904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.917350 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:40.951373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.951336 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr"] Nov 01 22:47:40.956805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.956781 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:40.961887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.961865 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e36490\"" Nov 01 22:47:40.963048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.963027 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e36490-dockercfg-bvrpr\"" Nov 01 22:47:40.969771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:40.969749 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr"] Nov 01 22:47:41.058977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.056910 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzxmm\" (UniqueName: \"kubernetes.io/projected/106ec4c2-48f8-43d6-9dec-64bd302aa012-kube-api-access-fzxmm\") pod \"payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:41.058977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.057233 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106ec4c2-48f8-43d6-9dec-64bd302aa012-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:41.058977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.058791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:41.221626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.198109 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106ec4c2-48f8-43d6-9dec-64bd302aa012-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:41.221626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.198337 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzxmm\" (UniqueName: \"kubernetes.io/projected/106ec4c2-48f8-43d6-9dec-64bd302aa012-kube-api-access-fzxmm\") pod \"payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:41.221626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.199550 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106ec4c2-48f8-43d6-9dec-64bd302aa012-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:41.315602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.271620 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d"] Nov 01 22:47:41.373846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.356365 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzxmm\" (UniqueName: \"kubernetes.io/projected/106ec4c2-48f8-43d6-9dec-64bd302aa012-kube-api-access-fzxmm\") pod \"payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:41.373846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.367076 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:41.481419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.481331 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:41.505602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.505551 4981 patch_prober.go:29] interesting pod/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:47:41.505868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.505839 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 22:47:41.557646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.557615 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:41.564834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.564783 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d83862c-2c75-4fd8-8034-cb297feb10ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:41.565393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.565356 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjc5\" (UniqueName: \"kubernetes.io/projected/9d83862c-2c75-4fd8-8034-cb297feb10ce-kube-api-access-xzjc5\") pod \"payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:41.622997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.573679 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d"] Nov 01 22:47:41.694794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.673655 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d83862c-2c75-4fd8-8034-cb297feb10ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:41.694794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.673783 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjc5\" (UniqueName: \"kubernetes.io/projected/9d83862c-2c75-4fd8-8034-cb297feb10ce-kube-api-access-xzjc5\") pod \"payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:41.702650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.699891 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d83862c-2c75-4fd8-8034-cb297feb10ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:41.753688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.741319 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjc5\" (UniqueName: \"kubernetes.io/projected/9d83862c-2c75-4fd8-8034-cb297feb10ce-kube-api-access-xzjc5\") pod \"payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:41.849958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.849113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:41.926601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.925698 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Nov 01 22:47:41.926601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.925761 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Nov 01 22:47:41.962037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.961997 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg"] Nov 01 22:47:41.962354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.962246 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" containerID="cri-o://6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d" gracePeriod=30 Nov 01 22:47:41.967187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:41.967157 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:42.177629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.177599 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:42.394151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.394001 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr"] Nov 01 22:47:42.405635 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:47:42.395017 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106ec4c2_48f8_43d6_9dec_64bd302aa012.slice/crio-01bd32f34cf26a442e7856b8c046f0666b8aacf2e8b54c4e79ededb749217e55 WatchSource:0}: Error finding container 01bd32f34cf26a442e7856b8c046f0666b8aacf2e8b54c4e79ededb749217e55: Status 404 returned error can't find the container with id 01bd32f34cf26a442e7856b8c046f0666b8aacf2e8b54c4e79ededb749217e55 Nov 01 22:47:42.491863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.477980 4981 patch_prober.go:29] interesting pod/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Nov 01 22:47:42.491863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.478043 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Nov 01 22:47:42.497773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.497706 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" event={"ID":"106ec4c2-48f8-43d6-9dec-64bd302aa012","Type":"ContainerStarted","Data":"01bd32f34cf26a442e7856b8c046f0666b8aacf2e8b54c4e79ededb749217e55"} Nov 01 22:47:42.755451 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:47:42.755415 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d83862c_2c75_4fd8_8034_cb297feb10ce.slice/crio-b37d001068841a8c875d87ccd82a334a10651ccf64cf572941f9ba6496005d6c WatchSource:0}: Error finding container b37d001068841a8c875d87ccd82a334a10651ccf64cf572941f9ba6496005d6c: Status 404 returned error can't find the container with id b37d001068841a8c875d87ccd82a334a10651ccf64cf572941f9ba6496005d6c Nov 01 22:47:42.755667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.755446 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d"] Nov 01 22:47:42.898569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.898537 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:47:42.899098 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.899073 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:47:42.899844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:42.899825 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:47:43.087567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.087495 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:47:43.179123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.179088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:43.481804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.481752 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:47:43.482200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.482182 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:47:43.482928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.482909 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:47:43.509395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.508902 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:43.509754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.509726 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" event={"ID":"106ec4c2-48f8-43d6-9dec-64bd302aa012","Type":"ContainerStarted","Data":"22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95"} Nov 01 22:47:43.511684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.511651 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" event={"ID":"9d83862c-2c75-4fd8-8034-cb297feb10ce","Type":"ContainerStarted","Data":"8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c"} Nov 01 22:47:43.511794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.511701 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" event={"ID":"9d83862c-2c75-4fd8-8034-cb297feb10ce","Type":"ContainerStarted","Data":"b37d001068841a8c875d87ccd82a334a10651ccf64cf572941f9ba6496005d6c"} Nov 01 22:47:43.511794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.511752 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:43.577478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.577428 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" podStartSLOduration=3.577413064 podStartE2EDuration="3.577413064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:43.559894324 +0000 UTC m=+1428.987772256" watchObservedRunningTime="2026-11-01 22:47:43.577413064 +0000 UTC m=+1429.005291016" Nov 01 22:47:43.603266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:43.598305 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" podStartSLOduration=3.598286269 podStartE2EDuration="3.598286269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:43.597529862 +0000 UTC m=+1429.025407795" watchObservedRunningTime="2026-11-01 22:47:43.598286269 +0000 UTC m=+1429.026164203" Nov 01 22:47:44.009981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:44.009949 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:44.529356 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:47:44.529320 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:47:44.763274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:44.763244 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:45.563076 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:45.563046 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:46.282392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:46.282342 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:47.215855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.215791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:47.329871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.329846 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:47:47.400295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400257 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-data\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400370 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-envoy\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400437 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-token\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-token\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400468 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-socket\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400651 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-data" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.400706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400686 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-certs\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400841 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-kube-api-access-6bqbr\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-kube-api-access-6bqbr\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400894 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-podinfo\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400951 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istiod-ca-cert\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.400994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.400980 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-credential-socket\") pod \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\" (UID: \"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8\") " Nov 01 22:47:47.402453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.401780 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istiod-ca-cert" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:47:47.402453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.402032 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-credential-socket" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.402453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.402175 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.402453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.402200 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.402453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.402214 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.402453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.402213 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-socket" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.402453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.402421 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-certs" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.403618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.403463 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-token" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:47.403618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.403501 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-envoy" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:47:47.408148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.407969 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-kube-api-access-6bqbr" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "kube-api-access-6bqbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:47:47.417363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.417334 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-podinfo" pod "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" (UID: "aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:47:47.504290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.504242 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.504290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.504274 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.504290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.504290 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.504537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.504303 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bqbr\" (UniqueName: \"kubernetes.io/projected/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-kube-api-access-6bqbr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.504537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.504316 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.504537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.504328 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:47:47.551282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.551200 4981 generic.go:356] "Generic (PLEG): container finished" podID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerID="6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d" exitCode=0 Nov 01 22:47:47.551282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.551263 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" Nov 01 22:47:47.551501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.551277 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" event={"ID":"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8","Type":"ContainerDied","Data":"6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d"} Nov 01 22:47:47.551501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.551341 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg" event={"ID":"aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8","Type":"ContainerDied","Data":"e9c087c84d699cbf3d4dfdead961890ca1410c3c1dc597b6b829c23f36211eba"} Nov 01 22:47:47.551501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.551367 4981 scope.go:122] "RemoveContainer" containerID="6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d" Nov 01 22:47:47.563211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.563193 4981 scope.go:122] "RemoveContainer" containerID="d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768" Nov 01 22:47:47.575522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.575503 4981 scope.go:122] "RemoveContainer" containerID="6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d" Nov 01 22:47:47.575870 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:47:47.575848 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d\": container with ID starting with 6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d not found: ID does not exist" containerID="6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d" Nov 01 22:47:47.575995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.575879 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d"} err="failed to get container status \"6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d\": rpc error: code = NotFound desc = could not find container \"6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d\": container with ID starting with 6d0ec13bb05668c8f607bd1f4311a8c37a0d8fbc8710987c6db6c06038e5917d not found: ID does not exist" Nov 01 22:47:47.575995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.575899 4981 scope.go:122] "RemoveContainer" containerID="d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768" Nov 01 22:47:47.576220 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:47:47.576196 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768\": container with ID starting with d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768 not found: ID does not exist" containerID="d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768" Nov 01 22:47:47.576269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.576228 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768"} err="failed to get container status \"d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768\": rpc error: code = NotFound desc = could not find container \"d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768\": container with ID starting with d327e7d665cd7ed3d5e2e92a88046fc664ea0a36184a8b0c34a6bb4d2e37a768 not found: ID does not exist" Nov 01 22:47:47.581119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.580825 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg"] Nov 01 22:47:47.587010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:47.586981 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-30578128-openshift-default-794c9bdc97-r7dmg"] Nov 01 22:47:48.124732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:48.120940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:48.737789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:48.737749 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" path="/var/lib/kubelet/pods/aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8/volumes" Nov 01 22:47:48.780873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:48.780841 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:49.549481 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:49.549446 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:50.187145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:50.187094 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:51.257467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:51.257432 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:51.348802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:51.348768 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:51.587345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:51.587260 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:52.460654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:52.460621 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:53.312158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:53.312112 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:54.075195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:54.075157 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:54.526069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:54.526034 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:47:54.908351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:54.908261 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:55.533654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:55.533627 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:47:55.762464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:55.762426 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:55.890547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:55.890455 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:56.640777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:56.640736 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:57.439668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:57.439633 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:57.608112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:57.608070 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:58.592995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:58.592948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:59.482064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:59.482028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:59.590833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:59.590784 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:59.792755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:59.792667 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:47:59.814830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:59.814803 4981 scope.go:122] "RemoveContainer" containerID="8ae8d1690ccc4d64c820c64031ff9c9d09672a08775e9a26d5763996ae65cd3a" Nov 01 22:47:59.843932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:59.843909 4981 scope.go:122] "RemoveContainer" containerID="9fdc7bd9b985c6429aa8c7678da9067fd10990f1a10a67f7093d8b391272b082" Nov 01 22:47:59.854288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:59.854268 4981 scope.go:122] "RemoveContainer" containerID="6125a205f272a8405f2ce53525cbf19be55df25c5310200405a0203424b0c1b9" Nov 01 22:47:59.864454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:47:59.864434 4981 scope.go:122] "RemoveContainer" containerID="06b05a5276823ecae11352f5a4fb37a87fc9db8bbd4992b9500836f4ef878ab5" Nov 01 22:48:00.506906 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:00.506871 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:01.265707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:01.265672 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:01.964553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:01.964503 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:02.191892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:02.191851 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:02.689936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:02.689900 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:02.890726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:02.890688 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:03.069069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.068982 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx"] Nov 01 22:48:03.070005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.069984 4981 cpu_manager.go:414] "containerMap: removing container" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" Nov 01 22:48:03.070070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.070009 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" Nov 01 22:48:03.070243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.070230 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" Nov 01 22:48:03.070286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.070248 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" Nov 01 22:48:03.076749 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.076725 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.087858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.087830 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f07902-openshift-default-dockercfg-lwgv9\"" Nov 01 22:48:03.095070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.095044 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx"] Nov 01 22:48:03.155598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.152235 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.230432 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-credential-socket\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.230534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-socket\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.230734 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d275241-c066-4ab8-828a-b6acab847630-istio-podinfo\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.230792 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-data\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.230874 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-certs\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.230904 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbfv6\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-kube-api-access-wbfv6\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.231082 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-istio-token\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.231139 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d275241-c066-4ab8-828a-b6acab847630-istiod-ca-cert\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.239646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.231234 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-envoy\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.462679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.462538 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d275241-c066-4ab8-828a-b6acab847630-istio-podinfo\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.463097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.463063 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-data\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.463321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.463302 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-certs\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.463440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.463426 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbfv6\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-kube-api-access-wbfv6\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.463744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.463711 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-istio-token\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.463957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.463929 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d275241-c066-4ab8-828a-b6acab847630-istiod-ca-cert\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.464166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.464146 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-envoy\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.464378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.464357 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-credential-socket\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.464689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.464644 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-socket\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.465432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.465409 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-socket\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.466611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.466545 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-data\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.473943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.466996 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-certs\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.474208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.468262 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-credential-socket\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.474342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.473748 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d275241-c066-4ab8-828a-b6acab847630-istiod-ca-cert\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.481282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.481013 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbfv6\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-kube-api-access-wbfv6\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.485883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.485848 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-istio-token\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.486797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.486770 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d275241-c066-4ab8-828a-b6acab847630-istio-podinfo\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.488724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.488674 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-envoy\") pod \"e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.531881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.531847 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:03.698013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.697974 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:03.959805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.959770 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:03.989763 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:03.989730 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d275241_c066_4ab8_828a_b6acab847630.slice/crio-3fbe34e69ffd2c511f26ec3ba98b875344033094d0be40421cd0115bfa941d28 WatchSource:0}: Error finding container 3fbe34e69ffd2c511f26ec3ba98b875344033094d0be40421cd0115bfa941d28: Status 404 returned error can't find the container with id 3fbe34e69ffd2c511f26ec3ba98b875344033094d0be40421cd0115bfa941d28 Nov 01 22:48:03.991626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.991601 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx"] Nov 01 22:48:03.992728 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.992698 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:03.992784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.992762 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:03.992817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:03.992795 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:04.801179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:04.801138 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" event={"ID":"2d275241-c066-4ab8-828a-b6acab847630","Type":"ContainerStarted","Data":"b4b94f042fb1eca9188b89ba18bf6bd6cba017a9ba67d22d51dc06c84300f8db"} Nov 01 22:48:04.801629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:04.801234 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" event={"ID":"2d275241-c066-4ab8-828a-b6acab847630","Type":"ContainerStarted","Data":"3fbe34e69ffd2c511f26ec3ba98b875344033094d0be40421cd0115bfa941d28"} Nov 01 22:48:04.833221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:04.833174 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podStartSLOduration=1.8331601530000001 podStartE2EDuration="1.833160153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:04.832832356 +0000 UTC m=+1450.260710288" watchObservedRunningTime="2026-11-01 22:48:04.833160153 +0000 UTC m=+1450.261038084" Nov 01 22:48:05.042651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:05.039276 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:05.698518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:05.698484 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:05.704169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:05.704143 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:05.809419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:05.809383 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:05.810353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:05.810336 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:48:05.991165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:05.991096 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:07.322599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:07.313266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:07.470598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:07.469885 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:07.619656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:07.618879 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:08.036600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.031832 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:48:08.296307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.296210 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.628856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.628759 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:48:08.630975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.629411 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.631398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.629446 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.631687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.629474 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:48:08.668602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.665651 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.690375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.690323 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:08.707611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.699462 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.729208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.729161 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.774559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.750549 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.776146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.750807 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.847601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.832656 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:08.847601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:08.834235 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:48:09.108600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.106151 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.108600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.106797 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:48:09.167599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.155917 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:48:09.302124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.302089 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:09.328300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.324302 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.329192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.329167 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:48:09.387692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.374005 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 01 22:48:09.671759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.671661 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.738316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.738261 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.740185 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.740501 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.743647 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.772191 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.772334 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.772467 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.757564 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.753855 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.775857 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.777155 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.754393 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.755732 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.784679 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.784771 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.784904 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.785037 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.785100 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:48:09.787877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.785234 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:48:09.897851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:09.894711 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:10.080077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.080040 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:48:10.107175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.055199 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:10.134867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.134830 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:48:10.135074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.134912 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf"] Nov 01 22:48:10.138156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.138116 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:10.172421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.172368 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:10.306259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.306215 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:10.376960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.355612 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:10.470672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.455411 4981 cpu_manager.go:414] "containerMap: removing container" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" Nov 01 22:48:10.470672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.455460 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae1ca54-b47f-49eb-af0a-a2b5b65cb8f8" containerName="istio-proxy" Nov 01 22:48:10.719599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.709278 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:10.816843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.816808 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:10.834379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.825058 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:48:10.860179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.825161 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:10.865764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832673 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 22:48:10.894601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832711 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:48:10.895016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832730 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832749 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832767 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832816 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832836 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.832855 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.878978 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.883908 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f07902\"" Nov 01 22:48:11.078651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:10.922083 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f07902-dockercfg-tfrnt\"" Nov 01 22:48:11.145297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.145216 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 01 22:48:11.158009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.157964 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:48:11.169751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.169712 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:48:11.250730 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.212505 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:48:11.250730 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.212566 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:11.369930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.354715 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:11.483079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.362744 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghb4\" (UniqueName: \"kubernetes.io/projected/44d06625-2be0-4199-b8bb-91fd93ef016a-kube-api-access-2ghb4\") pod \"payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:11.483079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.461531 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d06625-2be0-4199-b8bb-91fd93ef016a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:11.483079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.400965 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:11.483079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.395032 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vmhfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:48:11.483079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.469870 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:11.483079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.459622 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:11.508601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.401001 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:48:11.914867 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:48:11.914827 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.08s" Nov 01 22:48:11.931603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.918628 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf"] Nov 01 22:48:11.978698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.761415 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:11.984601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.595103 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d06625-2be0-4199-b8bb-91fd93ef016a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:11.984601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.979339 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghb4\" (UniqueName: \"kubernetes.io/projected/44d06625-2be0-4199-b8bb-91fd93ef016a-kube-api-access-2ghb4\") pod \"payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:11.984601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.965742 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:48:11.984601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.527479 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:12.000534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:11.978828 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:48:12.058606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.036328 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:12.082527 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.037707 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:48:12.097128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.037755 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:48:12.117814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.037782 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:48:12.119075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.037817 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:48:12.168713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.037846 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:12.168713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.037870 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:12.176651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.116224 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:12.204256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.204222 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:48:12.206188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.204162 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:48:12.325685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.325557 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d06625-2be0-4199-b8bb-91fd93ef016a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:12.469775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.469726 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:48:12.471712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.470358 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:48:12.473977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.473946 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghb4\" (UniqueName: \"kubernetes.io/projected/44d06625-2be0-4199-b8bb-91fd93ef016a-kube-api-access-2ghb4\") pod \"payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:12.576695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.546624 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:12.854896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.835845 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:48:12.854896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.847937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:13.005659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.990014 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.037750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.037698 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:48:13.041596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.992012 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:48:13.082598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:12.992043 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:48:13.154600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.086822 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.087495 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.124670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.124708 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.158644 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.158686 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.210680 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.212628 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.219383 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.237881 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.337094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.276799 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 01 22:48:13.475632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.475047 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:48:13.477182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.476655 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.625336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.527554 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:13.642131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.642055 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:13.867977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.843844 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:13.897641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.873658 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:14.083601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.846421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:14.083601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:13.990157 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:48:14.083601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.083189 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:48:14.094608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.020776 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:14.339930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.320852 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:14.339930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.326014 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:48:14.340332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.340294 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:48:14.361037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.352043 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:48:14.362647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.352362 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:48:14.363842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.352638 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:48:14.366857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.352854 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:14.369458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.354121 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:48:14.450132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.450094 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:14.536431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.507655 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:48:14.659861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.659724 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:48:14.706601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.670825 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded" Nov 01 22:48:15.067671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.708657 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:48:15.151870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.771320 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:15.182599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.951455 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:48:15.211514 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:48:15.197813 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:48:15.211836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.970328 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:15.212041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.030053 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:15.232598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.030090 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:48:15.273611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.096934 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:15.273611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.160228 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:15.273611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:14.970265 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 01 22:48:15.276316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.276281 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:15.481681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.479925 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:15.481681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.481148 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:48:15.481681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.481497 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:48:15.532678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.482058 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:48:15.532678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.482643 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:48:15.532678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.482920 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:48:15.532678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.483510 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:48:15.532678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.484147 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:15.532678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.484872 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:48:15.555096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.533281 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:15.719752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:15.674695 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:16.061606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:16.007218 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:17.214178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:16.380679 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:48:17.216001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:16.599712 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:17.216001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:16.853994 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:48:17.216001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:16.874788 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:17.216001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.041529 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 01 22:48:17.216285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.043880 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:48:17.218957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.043919 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:48:17.223781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.043938 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:48:17.223781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.051242 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:17.223992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.051271 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 01 22:48:17.223992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.071897 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:48:17.224961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.071935 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:48:17.227650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.071974 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:48:17.227650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.071995 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:48:17.227650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.072019 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:48:17.227650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.072054 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:48:17.231194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.072074 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:48:17.231925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.072094 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:48:17.307643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307610 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:48:17.307818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307698 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:48:17.307818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307731 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:48:17.307818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307764 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:48:17.307818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307798 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:48:17.307818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307814 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6"] Nov 01 22:48:17.308083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307849 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:48:17.308083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307880 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:48:17.308083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307909 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:48:17.308083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307937 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:48:17.308083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307968 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:48:17.308083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.307998 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:48:17.313465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.313369 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:48:17.315982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.315957 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2476f8-px2l6"] Nov 01 22:48:17.317603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.317523 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:48:17.323192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.323176 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:48:17.323361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.323343 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" event={"ID":"44d06625-2be0-4199-b8bb-91fd93ef016a","Type":"ContainerStarted","Data":"de163947ee4f051f175bd76eb85fb3101b751fb9be23bafa3c3c4eb4b6ba484d"} Nov 01 22:48:17.326324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.326303 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:48:17.345976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.345894 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf"] Nov 01 22:48:17.358599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.355823 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj"] Nov 01 22:48:17.364867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.364833 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbf233fc-b47tj"] Nov 01 22:48:17.432089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.432056 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:17.679049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:17.678942 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:18.335618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:18.335523 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" event={"ID":"44d06625-2be0-4199-b8bb-91fd93ef016a","Type":"ContainerStarted","Data":"e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199"} Nov 01 22:48:18.336258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:18.335789 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:18.353830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:18.353761 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" podStartSLOduration=9.353743063 podStartE2EDuration="9.353743063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:18.352284042 +0000 UTC m=+1463.780161977" watchObservedRunningTime="2026-11-01 22:48:18.353743063 +0000 UTC m=+1463.781620995" Nov 01 22:48:18.483744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:18.483709 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:18.741073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:18.741032 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eef9e5b-a10a-4bac-b2fa-0d483f069615" path="/var/lib/kubelet/pods/3eef9e5b-a10a-4bac-b2fa-0d483f069615/volumes" Nov 01 22:48:18.741766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:18.741745 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2fa6c6-4d86-418d-a5ce-ac6565a6820c" path="/var/lib/kubelet/pods/9d2fa6c6-4d86-418d-a5ce-ac6565a6820c/volumes" Nov 01 22:48:19.263230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:19.263193 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:19.488686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:19.488635 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:48:19.489178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:19.488699 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:19.835845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:19.835797 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:48:19.836064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:19.835877 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:48:19.919819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:19.919781 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:20.059082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.059040 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:20.225388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.221150 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:20.301858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.301825 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56"] Nov 01 22:48:20.303552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.303519 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" containerID="cri-o://32f85cd6c3dcc03eaf0e29ba0bdbf57b2f7dd2e23fb333355d6dac5216f1207c" gracePeriod=30 Nov 01 22:48:20.324838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.324778 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:20.537684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.533679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:20.791216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.791136 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7"] Nov 01 22:48:20.841602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.838642 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7"] Nov 01 22:48:20.841602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.838737 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.843100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.843078 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:20.843296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.843245 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:20.851463 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.851432 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.851698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.851677 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.851992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.851953 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5fc\" (UniqueName: \"kubernetes.io/projected/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-kube-api-access-zj5fc\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.874127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.874095 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:20.874272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.874257 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:48:20.956872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.953768 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.956872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.953839 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.956872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.954009 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5fc\" (UniqueName: \"kubernetes.io/projected/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-kube-api-access-zj5fc\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.956872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.955479 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.958837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.958806 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.970142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.970113 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5fc\" (UniqueName: \"kubernetes.io/projected/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-kube-api-access-zj5fc\") pod \"maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:20.997890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:20.989504 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:21.118646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:21.104664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:21.208615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:21.204202 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:21.569394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:21.569263 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:21.740987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:21.740820 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7"] Nov 01 22:48:21.746923 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:21.746889 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e8f96b_4bf9_4977_a86d_77aa8ba31f86.slice/crio-888e6110ca86e651bfbf57b0edf80db1d2c652e59136a8083d5c9d1cd0e74981 WatchSource:0}: Error finding container 888e6110ca86e651bfbf57b0edf80db1d2c652e59136a8083d5c9d1cd0e74981: Status 404 returned error can't find the container with id 888e6110ca86e651bfbf57b0edf80db1d2c652e59136a8083d5c9d1cd0e74981 Nov 01 22:48:22.539839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:22.539803 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" event={"ID":"77e8f96b-4bf9-4977-a86d-77aa8ba31f86","Type":"ContainerStarted","Data":"a61199bc8774e5f7e5d7b2a57364221d26821004ab2f3c0afd2c447bdaa2e44f"} Nov 01 22:48:22.540097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:22.539860 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" event={"ID":"77e8f96b-4bf9-4977-a86d-77aa8ba31f86","Type":"ContainerStarted","Data":"888e6110ca86e651bfbf57b0edf80db1d2c652e59136a8083d5c9d1cd0e74981"} Nov 01 22:48:22.563344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:22.563291 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" podStartSLOduration=2.563277659 podStartE2EDuration="2.563277659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:22.560453026 +0000 UTC m=+1467.988330979" watchObservedRunningTime="2026-11-01 22:48:22.563277659 +0000 UTC m=+1467.991155618" Nov 01 22:48:22.734207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:22.734152 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:23.611832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:23.551847 4981 generic.go:356] "Generic (PLEG): container finished" podID="77e8f96b-4bf9-4977-a86d-77aa8ba31f86" containerID="a61199bc8774e5f7e5d7b2a57364221d26821004ab2f3c0afd2c447bdaa2e44f" exitCode=0 Nov 01 22:48:23.611832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:23.551925 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" event={"ID":"77e8f96b-4bf9-4977-a86d-77aa8ba31f86","Type":"ContainerDied","Data":"a61199bc8774e5f7e5d7b2a57364221d26821004ab2f3c0afd2c447bdaa2e44f"} Nov 01 22:48:23.655829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:23.655801 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:24.463558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.463481 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:24.562809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.562777 4981 generic.go:356] "Generic (PLEG): container finished" podID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerID="8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928" exitCode=0 Nov 01 22:48:24.562984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.562860 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" event={"ID":"e49ed163-c875-4004-9f6e-3a5059d8ccf1","Type":"ContainerDied","Data":"8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928"} Nov 01 22:48:24.563517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.563497 4981 scope.go:122] "RemoveContainer" containerID="8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928" Nov 01 22:48:24.564888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.564868 4981 generic.go:356] "Generic (PLEG): container finished" podID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerID="3114dd87e4bdcffc34b982431cb285ff3f4c58026bb0a642aea0ac0255ca50a9" exitCode=0 Nov 01 22:48:24.564985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.564947 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" event={"ID":"b153d5e5-f84f-48a3-baea-50a3fbce1b12","Type":"ContainerDied","Data":"3114dd87e4bdcffc34b982431cb285ff3f4c58026bb0a642aea0ac0255ca50a9"} Nov 01 22:48:24.565348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.565331 4981 scope.go:122] "RemoveContainer" containerID="3114dd87e4bdcffc34b982431cb285ff3f4c58026bb0a642aea0ac0255ca50a9" Nov 01 22:48:24.565683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.565655 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.565761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.565731 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.565819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.565783 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.567066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.567046 4981 generic.go:356] "Generic (PLEG): container finished" podID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerID="cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463" exitCode=0 Nov 01 22:48:24.567168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.567119 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" event={"ID":"b1489db7-ad64-4845-8fdc-2474a55aaf0e","Type":"ContainerDied","Data":"cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463"} Nov 01 22:48:24.567280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.567257 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.567355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.567318 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.567422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.567357 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.567639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.567624 4981 scope.go:122] "RemoveContainer" containerID="cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463" Nov 01 22:48:24.570547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.570483 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.570657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.570563 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.570657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.570628 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:24.606623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:24.580681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:25.048881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.048854 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:48:25.189354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.188996 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:25.282692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.279635 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-xks-ca\") pod \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " Nov 01 22:48:25.282692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.279900 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-kube-api-access-zj5fc\" (UniqueName: \"kubernetes.io/projected/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-kube-api-access-zj5fc\") pod \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " Nov 01 22:48:25.282692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.280014 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-openshift-ca\") pod \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\" (UID: \"77e8f96b-4bf9-4977-a86d-77aa8ba31f86\") " Nov 01 22:48:25.282692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.282253 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-openshift-ca" pod "77e8f96b-4bf9-4977-a86d-77aa8ba31f86" (UID: "77e8f96b-4bf9-4977-a86d-77aa8ba31f86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:25.283561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.283515 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:25.301829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.301775 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-kube-api-access-zj5fc" pod "77e8f96b-4bf9-4977-a86d-77aa8ba31f86" (UID: "77e8f96b-4bf9-4977-a86d-77aa8ba31f86"). InnerVolumeSpecName "kube-api-access-zj5fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:25.303914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.303886 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-xks-ca" pod "77e8f96b-4bf9-4977-a86d-77aa8ba31f86" (UID: "77e8f96b-4bf9-4977-a86d-77aa8ba31f86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:25.387718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.387682 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj5fc\" (UniqueName: \"kubernetes.io/projected/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-kube-api-access-zj5fc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:25.387718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.387720 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e8f96b-4bf9-4977-a86d-77aa8ba31f86-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:25.724616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.723614 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" event={"ID":"b1489db7-ad64-4845-8fdc-2474a55aaf0e","Type":"ContainerStarted","Data":"2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7"} Nov 01 22:48:25.730250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.730220 4981 generic.go:356] "Generic (PLEG): container finished" podID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerID="32f85cd6c3dcc03eaf0e29ba0bdbf57b2f7dd2e23fb333355d6dac5216f1207c" exitCode=0 Nov 01 22:48:25.730520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.730499 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" event={"ID":"5a34390f-1892-407b-b5a6-bc3184b7bc09","Type":"ContainerDied","Data":"32f85cd6c3dcc03eaf0e29ba0bdbf57b2f7dd2e23fb333355d6dac5216f1207c"} Nov 01 22:48:25.730713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.730698 4981 scope.go:122] "RemoveContainer" containerID="e8683695c4b043a8ac74442d9fcc7499a54b39eacfbcd95b65c867a48b674f20" Nov 01 22:48:25.737506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.737469 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" event={"ID":"e49ed163-c875-4004-9f6e-3a5059d8ccf1","Type":"ContainerStarted","Data":"d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8"} Nov 01 22:48:25.741696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.741665 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" event={"ID":"77e8f96b-4bf9-4977-a86d-77aa8ba31f86","Type":"ContainerDied","Data":"888e6110ca86e651bfbf57b0edf80db1d2c652e59136a8083d5c9d1cd0e74981"} Nov 01 22:48:25.741941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.741888 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="888e6110ca86e651bfbf57b0edf80db1d2c652e59136a8083d5c9d1cd0e74981" Nov 01 22:48:25.742220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.742191 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7" Nov 01 22:48:25.755215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.755185 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" event={"ID":"b153d5e5-f84f-48a3-baea-50a3fbce1b12","Type":"ContainerStarted","Data":"4263a5d250ec163e44029a2c30fdb28464cbdd62fca1bc21ddbef6f563920a2c"} Nov 01 22:48:25.847632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:25.841899 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:26.035243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.035180 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:48:26.036056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.036027 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:26.036292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.036259 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:26.043212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.043130 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:48:26.165561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.165531 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-certs\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.165752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.165569 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-envoy\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.165752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.165652 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-kube-api-access-t45x5\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-kube-api-access-t45x5\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.165846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.165764 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a34390f-1892-407b-b5a6-bc3184b7bc09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34390f-1892-407b-b5a6-bc3184b7bc09-istiod-ca-cert\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.165846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.165786 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-token\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-token\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.165846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.165831 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-podinfo\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.166060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.166027 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-certs" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:26.166272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.166253 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-credential-socket\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.166395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.166376 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-data\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.166654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.166629 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a34390f-1892-407b-b5a6-bc3184b7bc09-istiod-ca-cert" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:26.166783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.166718 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-credential-socket" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:26.166985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.166967 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-socket\") pod \"5a34390f-1892-407b-b5a6-bc3184b7bc09\" (UID: \"5a34390f-1892-407b-b5a6-bc3184b7bc09\") " Nov 01 22:48:26.167198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.166994 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-data" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:26.167277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.167239 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-socket" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:26.168353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.168329 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.168486 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.168473 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34390f-1892-407b-b5a6-bc3184b7bc09-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.168602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.168589 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.168717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.168693 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.168717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.168717 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.168877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.168783 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-podinfo" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:48:26.169009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.168985 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-token" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:26.170283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.170258 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-kube-api-access-t45x5" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "kube-api-access-t45x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:26.170455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.170429 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-envoy" pod "5a34390f-1892-407b-b5a6-bc3184b7bc09" (UID: "5a34390f-1892-407b-b5a6-bc3184b7bc09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:48:26.271100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.271055 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t45x5\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-kube-api-access-t45x5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.271100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.271094 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.271328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.271109 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.271328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.271124 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34390f-1892-407b-b5a6-bc3184b7bc09-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:26.429936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.429896 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:48:26.431917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.431885 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:48:26.432182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.432154 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:48:26.567600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.566644 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:48:26.567600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.566796 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 22:48:26.567600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.566848 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 22:48:26.658600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:26.658408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:27.261555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.261513 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:27.284336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.284287 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:27.309690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.289871 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:27.475599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.457005 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:27.529516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.529418 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:48:27.584765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.584715 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:48:27.585434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.581766 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 22:48:27.627606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.626930 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 22:48:27.760228 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:48:27.756764 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.026s" Nov 01 22:48:27.766531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.766502 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f"] Nov 01 22:48:27.767744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.767491 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" Nov 01 22:48:27.767903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.767851 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" Nov 01 22:48:27.767980 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.767940 4981 cpu_manager.go:414] "containerMap: removing container" podUID="77e8f96b-4bf9-4977-a86d-77aa8ba31f86" containerName="revoke-keys" Nov 01 22:48:27.767980 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.767951 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e8f96b-4bf9-4977-a86d-77aa8ba31f86" containerName="revoke-keys" Nov 01 22:48:27.767980 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.756977 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" Nov 01 22:48:27.768133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.767988 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" Nov 01 22:48:27.768133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.767996 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" Nov 01 22:48:27.768496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.768471 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" Nov 01 22:48:27.768566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.768510 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" containerName="istio-proxy" Nov 01 22:48:27.768566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.768534 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e8f96b-4bf9-4977-a86d-77aa8ba31f86" containerName="revoke-keys" Nov 01 22:48:27.851603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.849000 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56" event={"ID":"5a34390f-1892-407b-b5a6-bc3184b7bc09","Type":"ContainerDied","Data":"abf08e6272726bed030878cd5db7f4cdb7c4228906dc19f94f3a27b7cfa16d96"} Nov 01 22:48:27.851603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.849062 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f"] Nov 01 22:48:27.851603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.849136 4981 scope.go:122] "RemoveContainer" containerID="32f85cd6c3dcc03eaf0e29ba0bdbf57b2f7dd2e23fb333355d6dac5216f1207c" Nov 01 22:48:27.851603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.849353 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:27.854425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.853867 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0e716d\"" Nov 01 22:48:27.961600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.957198 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txxm\" (UniqueName: \"kubernetes.io/projected/b030a076-acd3-4f9a-b258-770566ec72b6-kube-api-access-6txxm\") pod \"payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:27.961600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.957297 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b030a076-acd3-4f9a-b258-770566ec72b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:27.962705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.962679 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56"] Nov 01 22:48:27.966437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:27.966413 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2476f8-openshift-default-8489fc489f-79b56"] Nov 01 22:48:28.054652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.053724 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:28.054652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.053785 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:28.065562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.058882 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txxm\" (UniqueName: \"kubernetes.io/projected/b030a076-acd3-4f9a-b258-770566ec72b6-kube-api-access-6txxm\") pod \"payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:28.065562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.058969 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b030a076-acd3-4f9a-b258-770566ec72b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:28.065562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.060279 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b030a076-acd3-4f9a-b258-770566ec72b6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:28.070444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.070420 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txxm\" (UniqueName: \"kubernetes.io/projected/b030a076-acd3-4f9a-b258-770566ec72b6-kube-api-access-6txxm\") pod \"payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:28.164993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.164957 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:28.213667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.213634 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:28.391600 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:48:28.391205 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a34390f_1892_407b_b5a6_bc3184b7bc09.slice/crio-abf08e6272726bed030878cd5db7f4cdb7c4228906dc19f94f3a27b7cfa16d96\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a34390f_1892_407b_b5a6_bc3184b7bc09.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:48:28.431494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.431038 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:48:28.431494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.431102 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:48:28.559833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.559794 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 22:48:28.560078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.560053 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 22:48:28.772052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:28.772014 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a34390f-1892-407b-b5a6-bc3184b7bc09" path="/var/lib/kubelet/pods/5a34390f-1892-407b-b5a6-bc3184b7bc09/volumes" Nov 01 22:48:29.056655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.055820 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:29.056655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.055897 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:29.157603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.155404 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f"] Nov 01 22:48:29.172487 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:29.172108 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb030a076_acd3_4f9a_b258_770566ec72b6.slice/crio-90d17d8d5be003b748f2b763b4a999d773ad78342f268986780df65ad021a5f7 WatchSource:0}: Error finding container 90d17d8d5be003b748f2b763b4a999d773ad78342f268986780df65ad021a5f7: Status 404 returned error can't find the container with id 90d17d8d5be003b748f2b763b4a999d773ad78342f268986780df65ad021a5f7 Nov 01 22:48:29.260613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.238033 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:29.445613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.433289 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:48:29.447714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.447671 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:48:29.449048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.449030 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:48:29.559412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.557189 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 22:48:29.559412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.557244 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 22:48:29.819639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.799667 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" event={"ID":"b030a076-acd3-4f9a-b258-770566ec72b6","Type":"ContainerStarted","Data":"11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f"} Nov 01 22:48:29.819928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.819896 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" event={"ID":"b030a076-acd3-4f9a-b258-770566ec72b6","Type":"ContainerStarted","Data":"90d17d8d5be003b748f2b763b4a999d773ad78342f268986780df65ad021a5f7"} Nov 01 22:48:29.821508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.821488 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:29.896119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.896041 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" podStartSLOduration=3.896016549 podStartE2EDuration="3.896016549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:29.869387007 +0000 UTC m=+1475.297264938" watchObservedRunningTime="2026-11-01 22:48:29.896016549 +0000 UTC m=+1475.323894481" Nov 01 22:48:29.969054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:29.968992 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:30.057608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.048174 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:30.057926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.057888 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:30.329652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.329152 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:30.450629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.448356 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:48:30.450629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.448440 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:48:30.559068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.559034 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 22:48:30.564670 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.559297 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 22:48:30.869408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:30.869368 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:31.037610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.034990 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:31.037610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.035052 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:31.144023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.143933 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk"] Nov 01 22:48:31.230743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.229330 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.268615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.251715 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:48:31.268615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.251915 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:31.268615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.251992 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:31.268615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.252599 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:31.363652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.363623 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk"] Nov 01 22:48:31.430654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.422967 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.430654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.423125 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd25q\" (UniqueName: \"kubernetes.io/projected/e57ef9d0-23ad-4885-87c7-9fe034425a10-kube-api-access-wd25q\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.430654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.423218 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.463423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.462386 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:48:31.463423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.462445 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:48:31.528601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.528088 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd25q\" (UniqueName: \"kubernetes.io/projected/e57ef9d0-23ad-4885-87c7-9fe034425a10-kube-api-access-wd25q\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.528601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.528203 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.528601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.528530 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.532667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.529883 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.548611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.543805 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.558700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.556467 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 22:48:31.558700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.556528 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 22:48:31.559907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.559873 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd25q\" (UniqueName: \"kubernetes.io/projected/e57ef9d0-23ad-4885-87c7-9fe034425a10-kube-api-access-wd25q\") pod \"maas-api-revoke-keys-e2e-api-a-b34688-552pk\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.735656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.730443 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk"] Nov 01 22:48:31.759824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.759779 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4"] Nov 01 22:48:31.766899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.766867 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:31.833601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:31.820973 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:32.388602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.369117 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:32.388602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.369176 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:32.472849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.472755 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:48:32.492251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.485203 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:48:32.492251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.485147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:32.583181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.563961 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 01 22:48:32.583181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.564015 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 01 22:48:32.670611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.652427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:32.914025 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:32.913112 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:33.037666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.037631 4981 patch_prober.go:29] interesting pod/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 22:48:33.037918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.037890 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 22:48:33.123665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.121861 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:33.215730 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:33.215700 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57ef9d0_23ad_4885_87c7_9fe034425a10.slice/crio-8b6ed3643764dcf64bee42ddaaacf691d73ae65387dea436dcae88cdbcb8b5e1 WatchSource:0}: Error finding container 8b6ed3643764dcf64bee42ddaaacf691d73ae65387dea436dcae88cdbcb8b5e1: Status 404 returned error can't find the container with id 8b6ed3643764dcf64bee42ddaaacf691d73ae65387dea436dcae88cdbcb8b5e1 Nov 01 22:48:33.215873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.215770 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk"] Nov 01 22:48:33.435000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.434973 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:48:33.435448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.435431 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:48:33.436015 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.435996 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:48:33.447675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.447644 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:48:33.674010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.667544 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:48:33.712537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.712501 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" event={"ID":"e57ef9d0-23ad-4885-87c7-9fe034425a10","Type":"ContainerStarted","Data":"8b6ed3643764dcf64bee42ddaaacf691d73ae65387dea436dcae88cdbcb8b5e1"} Nov 01 22:48:33.714813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:33.714790 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:48:34.043165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:34.043060 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:48:34.044344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:34.044309 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:48:34.060709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:34.057452 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:48:34.064953 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:34.061334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:34.677530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:34.677504 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:34.720951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:34.720925 4981 generic.go:356] "Generic (PLEG): container finished" podID="e57ef9d0-23ad-4885-87c7-9fe034425a10" containerID="edb2a29755adfbf3f97bfcd4f0f9d81324ceeabc6e33c89f9315d4d5ca2d5254" exitCode=0 Nov 01 22:48:34.721092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:34.721015 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" event={"ID":"e57ef9d0-23ad-4885-87c7-9fe034425a10","Type":"ContainerDied","Data":"edb2a29755adfbf3f97bfcd4f0f9d81324ceeabc6e33c89f9315d4d5ca2d5254"} Nov 01 22:48:35.379552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:35.379524 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:35.916296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:35.916269 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:36.014661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.014557 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-xks-ca\") pod \"e57ef9d0-23ad-4885-87c7-9fe034425a10\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " Nov 01 22:48:36.014819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.014716 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57ef9d0-23ad-4885-87c7-9fe034425a10-kube-api-access-wd25q\" (UniqueName: \"kubernetes.io/projected/e57ef9d0-23ad-4885-87c7-9fe034425a10-kube-api-access-wd25q\") pod \"e57ef9d0-23ad-4885-87c7-9fe034425a10\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " Nov 01 22:48:36.014819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.014757 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-openshift-ca\") pod \"e57ef9d0-23ad-4885-87c7-9fe034425a10\" (UID: \"e57ef9d0-23ad-4885-87c7-9fe034425a10\") " Nov 01 22:48:36.015155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.015130 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-openshift-ca" pod "e57ef9d0-23ad-4885-87c7-9fe034425a10" (UID: "e57ef9d0-23ad-4885-87c7-9fe034425a10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:36.015403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.015385 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:36.016557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.016533 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-xks-ca" pod "e57ef9d0-23ad-4885-87c7-9fe034425a10" (UID: "e57ef9d0-23ad-4885-87c7-9fe034425a10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:36.016800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.016786 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57ef9d0-23ad-4885-87c7-9fe034425a10-kube-api-access-wd25q" pod "e57ef9d0-23ad-4885-87c7-9fe034425a10" (UID: "e57ef9d0-23ad-4885-87c7-9fe034425a10"). InnerVolumeSpecName "kube-api-access-wd25q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:36.069269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.069212 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:36.120321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.120248 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57ef9d0-23ad-4885-87c7-9fe034425a10-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:36.120321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.120292 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd25q\" (UniqueName: \"kubernetes.io/projected/e57ef9d0-23ad-4885-87c7-9fe034425a10-kube-api-access-wd25q\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:36.730922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.730880 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" podUID="2a2b93ab-425b-4070-b572-93e16f27f2e8" containerName="payload-processing" containerID="cri-o://a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d" gracePeriod=25 Nov 01 22:48:36.747170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.747139 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" event={"ID":"e57ef9d0-23ad-4885-87c7-9fe034425a10","Type":"ContainerDied","Data":"8b6ed3643764dcf64bee42ddaaacf691d73ae65387dea436dcae88cdbcb8b5e1"} Nov 01 22:48:36.747335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.747184 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b6ed3643764dcf64bee42ddaaacf691d73ae65387dea436dcae88cdbcb8b5e1" Nov 01 22:48:36.747335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.747184 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk" Nov 01 22:48:36.760683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.760653 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" podUID="814ffa64-0d94-4ace-8fc3-0ce77cecb94b" containerName="payload-pre-processing" containerID="cri-o://15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d" gracePeriod=25 Nov 01 22:48:36.810877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:36.810848 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:37.079364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.079332 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:48:37.137941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.137904 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2b93ab-425b-4070-b572-93e16f27f2e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2b93ab-425b-4070-b572-93e16f27f2e8-plugins-config-volume\") pod \"2a2b93ab-425b-4070-b572-93e16f27f2e8\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " Nov 01 22:48:37.138141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.138021 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2b93ab-425b-4070-b572-93e16f27f2e8-kube-api-access-k7qt4\" (UniqueName: \"kubernetes.io/projected/2a2b93ab-425b-4070-b572-93e16f27f2e8-kube-api-access-k7qt4\") pod \"2a2b93ab-425b-4070-b572-93e16f27f2e8\" (UID: \"2a2b93ab-425b-4070-b572-93e16f27f2e8\") " Nov 01 22:48:37.138358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.138331 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2b93ab-425b-4070-b572-93e16f27f2e8-plugins-config-volume" pod "2a2b93ab-425b-4070-b572-93e16f27f2e8" (UID: "2a2b93ab-425b-4070-b572-93e16f27f2e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:37.139371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.139333 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a2b93ab-425b-4070-b572-93e16f27f2e8-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:37.141069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.141044 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2b93ab-425b-4070-b572-93e16f27f2e8-kube-api-access-k7qt4" pod "2a2b93ab-425b-4070-b572-93e16f27f2e8" (UID: "2a2b93ab-425b-4070-b572-93e16f27f2e8"). InnerVolumeSpecName "kube-api-access-k7qt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:37.184109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.184081 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:48:37.240667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.240634 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-kube-api-access-tn2nr\" (UniqueName: \"kubernetes.io/projected/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-kube-api-access-tn2nr\") pod \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " Nov 01 22:48:37.240840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.240755 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-plugins-config-volume\") pod \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\" (UID: \"814ffa64-0d94-4ace-8fc3-0ce77cecb94b\") " Nov 01 22:48:37.241253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.241222 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-plugins-config-volume" pod "814ffa64-0d94-4ace-8fc3-0ce77cecb94b" (UID: "814ffa64-0d94-4ace-8fc3-0ce77cecb94b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:37.241781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.241748 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qt4\" (UniqueName: \"kubernetes.io/projected/2a2b93ab-425b-4070-b572-93e16f27f2e8-kube-api-access-k7qt4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:37.241899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.241781 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:37.243111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.243088 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-kube-api-access-tn2nr" pod "814ffa64-0d94-4ace-8fc3-0ce77cecb94b" (UID: "814ffa64-0d94-4ace-8fc3-0ce77cecb94b"). InnerVolumeSpecName "kube-api-access-tn2nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:37.347032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.346940 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2nr\" (UniqueName: \"kubernetes.io/projected/814ffa64-0d94-4ace-8fc3-0ce77cecb94b-kube-api-access-tn2nr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:37.599338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.599245 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.759448 4981 generic.go:356] "Generic (PLEG): container finished" podID="814ffa64-0d94-4ace-8fc3-0ce77cecb94b" containerID="15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d" exitCode=0 Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.759646 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.759664 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" event={"ID":"814ffa64-0d94-4ace-8fc3-0ce77cecb94b","Type":"ContainerDied","Data":"15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d"} Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.759745 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4" event={"ID":"814ffa64-0d94-4ace-8fc3-0ce77cecb94b","Type":"ContainerDied","Data":"30c5c3363c58d4b0fef1d03f96d9feefd812c6a04fd0333e24fb38d2468fc064"} Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.759780 4981 scope.go:122] "RemoveContainer" containerID="15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d" Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.761490 4981 generic.go:356] "Generic (PLEG): container finished" podID="2a2b93ab-425b-4070-b572-93e16f27f2e8" containerID="a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d" exitCode=0 Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.761545 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" Nov 01 22:48:37.761629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.761546 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" event={"ID":"2a2b93ab-425b-4070-b572-93e16f27f2e8","Type":"ContainerDied","Data":"a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d"} Nov 01 22:48:37.762215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.761655 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk" event={"ID":"2a2b93ab-425b-4070-b572-93e16f27f2e8","Type":"ContainerDied","Data":"a1d68d79c8059f0d40fc32fee6be032e678a7d275f8198a57a74c83a71812f6c"} Nov 01 22:48:37.768881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.768793 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:37.784156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.784132 4981 scope.go:122] "RemoveContainer" containerID="15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d" Nov 01 22:48:37.784537 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:48:37.784506 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d\": container with ID starting with 15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d not found: ID does not exist" containerID="15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d" Nov 01 22:48:37.784680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.784546 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d"} err="failed to get container status \"15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d\": rpc error: code = NotFound desc = could not find container \"15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d\": container with ID starting with 15b6850ea3da868fd00731fa51c82bf17226cb99184700264e9bbf78fde36e5d not found: ID does not exist" Nov 01 22:48:37.784680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.784568 4981 scope.go:122] "RemoveContainer" containerID="a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d" Nov 01 22:48:37.797964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.797945 4981 scope.go:122] "RemoveContainer" containerID="a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d" Nov 01 22:48:37.798318 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:48:37.798288 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d\": container with ID starting with a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d not found: ID does not exist" containerID="a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d" Nov 01 22:48:37.798439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.798322 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d"} err="failed to get container status \"a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d\": rpc error: code = NotFound desc = could not find container \"a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d\": container with ID starting with a99dae14add3b413dcef581a95393e4c9ccfe36cdb22bd761d6449d0c802f32d not found: ID does not exist" Nov 01 22:48:37.800917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.800897 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4"] Nov 01 22:48:37.806358 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.806334 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c0d076-6cc476c575-92ph4"] Nov 01 22:48:37.814849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.814831 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk"] Nov 01 22:48:37.820342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:37.820324 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c0d076-5b556b7b7c-db8mk"] Nov 01 22:48:38.186102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.186065 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw"] Nov 01 22:48:38.187319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187179 4981 cpu_manager.go:414] "containerMap: removing container" podUID="814ffa64-0d94-4ace-8fc3-0ce77cecb94b" containerName="payload-pre-processing" Nov 01 22:48:38.187319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187201 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814ffa64-0d94-4ace-8fc3-0ce77cecb94b" containerName="payload-pre-processing" Nov 01 22:48:38.187319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187229 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e57ef9d0-23ad-4885-87c7-9fe034425a10" containerName="revoke-keys" Nov 01 22:48:38.187319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187237 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57ef9d0-23ad-4885-87c7-9fe034425a10" containerName="revoke-keys" Nov 01 22:48:38.187319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187273 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2a2b93ab-425b-4070-b572-93e16f27f2e8" containerName="payload-processing" Nov 01 22:48:38.187319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187281 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2b93ab-425b-4070-b572-93e16f27f2e8" containerName="payload-processing" Nov 01 22:48:38.187714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187512 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="814ffa64-0d94-4ace-8fc3-0ce77cecb94b" containerName="payload-pre-processing" Nov 01 22:48:38.187714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187531 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57ef9d0-23ad-4885-87c7-9fe034425a10" containerName="revoke-keys" Nov 01 22:48:38.187714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.187543 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2b93ab-425b-4070-b572-93e16f27f2e8" containerName="payload-processing" Nov 01 22:48:38.193183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.193160 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.206733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.206698 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b78bd-openshift-default-dockercfg-hv5s7\"" Nov 01 22:48:38.211258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.211232 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw"] Nov 01 22:48:38.238887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.238850 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.280426 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.280489 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.280545 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.280660 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.280761 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.280831 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lfzr\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-kube-api-access-9lfzr\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.280916 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.281021 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.290941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.281128 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.384734 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.384847 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.384945 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.385017 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.385046 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.385096 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.385186 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.385288 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.385596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.385361 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lfzr\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-kube-api-access-9lfzr\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.386236 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.386148 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.389699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.386370 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.389699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.386908 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.389699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.387886 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.389699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.388407 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.451752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.446874 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.451752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.447176 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.472595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.459432 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lfzr\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-kube-api-access-9lfzr\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.501631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.500223 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.514627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.514334 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:38.738074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.737992 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:38.743547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.743520 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2b93ab-425b-4070-b572-93e16f27f2e8" path="/var/lib/kubelet/pods/2a2b93ab-425b-4070-b572-93e16f27f2e8/volumes" Nov 01 22:48:38.744400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.744378 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814ffa64-0d94-4ace-8fc3-0ce77cecb94b" path="/var/lib/kubelet/pods/814ffa64-0d94-4ace-8fc3-0ce77cecb94b/volumes" Nov 01 22:48:38.850652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.850619 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:38.943673 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:38.943640 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb000cee6_3d32_42de_ad5d_fa3d0e4fbb61.slice/crio-dc6f47dcdc81ddbeed3dcc452be36501e0262e69561fc3eb53f3bfba809a2bd1 WatchSource:0}: Error finding container dc6f47dcdc81ddbeed3dcc452be36501e0262e69561fc3eb53f3bfba809a2bd1: Status 404 returned error can't find the container with id dc6f47dcdc81ddbeed3dcc452be36501e0262e69561fc3eb53f3bfba809a2bd1 Nov 01 22:48:38.946334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.945934 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw"] Nov 01 22:48:38.948124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.948054 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:38.948263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.948140 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:38.948263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:38.948187 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:39.826166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:39.826123 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:39.871227 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:39.871186 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" event={"ID":"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61","Type":"ContainerStarted","Data":"cb9dccbfbac2618181f24c84af91df0b6beecd726fe630392cf5847807901a4d"} Nov 01 22:48:39.871507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:39.871486 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" event={"ID":"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61","Type":"ContainerStarted","Data":"dc6f47dcdc81ddbeed3dcc452be36501e0262e69561fc3eb53f3bfba809a2bd1"} Nov 01 22:48:39.944183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:39.929132 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podStartSLOduration=1.929108377 podStartE2EDuration="1.929108377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:39.926938719 +0000 UTC m=+1485.354816651" watchObservedRunningTime="2026-11-01 22:48:39.929108377 +0000 UTC m=+1485.356986309" Nov 01 22:48:40.521635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:40.518745 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:40.527788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:40.527756 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:40.898658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:40.898210 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:40.912613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:40.903939 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:48:41.235744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:41.235628 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:41.564773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:41.564662 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:41.665503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:41.665473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:42.510041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:42.510010 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:48:42.703834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:42.703792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:42.890381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:42.884478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:44.010592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.009817 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb"] Nov 01 22:48:44.029450 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.029411 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:44.037886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.037852 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb"] Nov 01 22:48:44.038073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.037991 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.041178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.041141 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f"] Nov 01 22:48:44.048099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.047777 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.055762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.055312 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6b78bd-dockercfg-tlxkf\"" Nov 01 22:48:44.055762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.055320 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6b78bd\"" Nov 01 22:48:44.067138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.066849 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f"] Nov 01 22:48:44.183597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.183012 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb05797-6bbf-45c1-a341-088aeadb04e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.183597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.183096 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f207d6-ed80-4b74-a330-57b836b5fba5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.183597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.183418 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5h95\" (UniqueName: \"kubernetes.io/projected/15f207d6-ed80-4b74-a330-57b836b5fba5-kube-api-access-l5h95\") pod \"payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.183597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.183511 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz62b\" (UniqueName: \"kubernetes.io/projected/2eb05797-6bbf-45c1-a341-088aeadb04e0-kube-api-access-dz62b\") pod \"payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.296653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.289712 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5h95\" (UniqueName: \"kubernetes.io/projected/15f207d6-ed80-4b74-a330-57b836b5fba5-kube-api-access-l5h95\") pod \"payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.296653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.289831 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz62b\" (UniqueName: \"kubernetes.io/projected/2eb05797-6bbf-45c1-a341-088aeadb04e0-kube-api-access-dz62b\") pod \"payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.296653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.289976 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb05797-6bbf-45c1-a341-088aeadb04e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.296653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.290025 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f207d6-ed80-4b74-a330-57b836b5fba5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.341598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.300558 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb05797-6bbf-45c1-a341-088aeadb04e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.341598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.315525 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f207d6-ed80-4b74-a330-57b836b5fba5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.341598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.319380 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5h95\" (UniqueName: \"kubernetes.io/projected/15f207d6-ed80-4b74-a330-57b836b5fba5-kube-api-access-l5h95\") pod \"payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.341598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.332311 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz62b\" (UniqueName: \"kubernetes.io/projected/2eb05797-6bbf-45c1-a341-088aeadb04e0-kube-api-access-dz62b\") pod \"payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.415049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.378940 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:44.416289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.415905 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:44.446601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.431119 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:44.690610 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:48:44.674037 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:48:44.727487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:44.727325 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:45.014103 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:45.014068 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f207d6_ed80_4b74_a330_57b836b5fba5.slice/crio-b7ca5dcc747c8c5f82bf3069a335de39703ed5c3424e4e8b997558218d5e208b WatchSource:0}: Error finding container b7ca5dcc747c8c5f82bf3069a335de39703ed5c3424e4e8b997558218d5e208b: Status 404 returned error can't find the container with id b7ca5dcc747c8c5f82bf3069a335de39703ed5c3424e4e8b997558218d5e208b Nov 01 22:48:45.014564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.014353 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f"] Nov 01 22:48:45.045368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.045337 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" event={"ID":"15f207d6-ed80-4b74-a330-57b836b5fba5","Type":"ContainerStarted","Data":"b7ca5dcc747c8c5f82bf3069a335de39703ed5c3424e4e8b997558218d5e208b"} Nov 01 22:48:45.058162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.058135 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr"] Nov 01 22:48:45.067879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.067684 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.072662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.072637 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:48:45.073546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.073528 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:48:45.097650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.073919 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:48:45.097650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.078638 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:48:45.107135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.107108 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr"] Nov 01 22:48:45.154422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.154397 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb"] Nov 01 22:48:45.211346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.211315 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bd4957-9025-48d7-adde-2d015113976c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.211481 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.211432 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bd4957-9025-48d7-adde-2d015113976c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.211551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.211513 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbs7\" (UniqueName: \"kubernetes.io/projected/26bd4957-9025-48d7-adde-2d015113976c-kube-api-access-kqbs7\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.313172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.313141 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bd4957-9025-48d7-adde-2d015113976c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.313323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.313209 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bd4957-9025-48d7-adde-2d015113976c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.313323 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.313243 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbs7\" (UniqueName: \"kubernetes.io/projected/26bd4957-9025-48d7-adde-2d015113976c-kube-api-access-kqbs7\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.313991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.313969 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bd4957-9025-48d7-adde-2d015113976c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.317590 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.316294 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bd4957-9025-48d7-adde-2d015113976c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.324647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.324551 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbs7\" (UniqueName: \"kubernetes.io/projected/26bd4957-9025-48d7-adde-2d015113976c-kube-api-access-kqbs7\") pod \"maas-api-revoke-sub-0d7047c2-x6ntr\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.415231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.415146 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:45.506885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.506820 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:45.727457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:45.727427 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr"] Nov 01 22:48:45.728667 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:45.728640 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bd4957_9025_48d7_adde_2d015113976c.slice/crio-284a64a7a5221a726807c18bc35649b50a3ffd823bf798dee62809e80e099912 WatchSource:0}: Error finding container 284a64a7a5221a726807c18bc35649b50a3ffd823bf798dee62809e80e099912: Status 404 returned error can't find the container with id 284a64a7a5221a726807c18bc35649b50a3ffd823bf798dee62809e80e099912 Nov 01 22:48:46.055547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.055512 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" event={"ID":"2eb05797-6bbf-45c1-a341-088aeadb04e0","Type":"ContainerStarted","Data":"0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564"} Nov 01 22:48:46.055994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.055571 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" event={"ID":"2eb05797-6bbf-45c1-a341-088aeadb04e0","Type":"ContainerStarted","Data":"1380d829cdda838ba3ac1a34583b28e2e14d6d75d97813f9713539c7d622654e"} Nov 01 22:48:46.055994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.055611 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:46.057170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.057144 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" event={"ID":"26bd4957-9025-48d7-adde-2d015113976c","Type":"ContainerStarted","Data":"284a64a7a5221a726807c18bc35649b50a3ffd823bf798dee62809e80e099912"} Nov 01 22:48:46.058728 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.058682 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" event={"ID":"15f207d6-ed80-4b74-a330-57b836b5fba5","Type":"ContainerStarted","Data":"85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7"} Nov 01 22:48:46.058914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.058800 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:46.097599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.089014 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" podStartSLOduration=3.088995069 podStartE2EDuration="3.088995069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:46.08394242 +0000 UTC m=+1491.511820354" watchObservedRunningTime="2026-11-01 22:48:46.088995069 +0000 UTC m=+1491.516873004" Nov 01 22:48:46.118238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.118170 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" podStartSLOduration=3.118152297 podStartE2EDuration="3.118152297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:46.11719407 +0000 UTC m=+1491.545072014" watchObservedRunningTime="2026-11-01 22:48:46.118152297 +0000 UTC m=+1491.546030229" Nov 01 22:48:46.204604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.203738 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:46.330889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.330848 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:46.525655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:46.525619 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:47.112477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:47.112377 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" event={"ID":"26bd4957-9025-48d7-adde-2d015113976c","Type":"ContainerStarted","Data":"8429056fc870c1bfe0fcecb6a71986b54e0b6005308b1fb61aeb241ff06df78f"} Nov 01 22:48:47.141040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:47.140965 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" podStartSLOduration=2.140946026 podStartE2EDuration="2.140946026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:47.138292186 +0000 UTC m=+1492.566170190" watchObservedRunningTime="2026-11-01 22:48:47.140946026 +0000 UTC m=+1492.568823958" Nov 01 22:48:47.389901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:47.389822 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:48.123123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:48.123087 4981 generic.go:356] "Generic (PLEG): container finished" podID="26bd4957-9025-48d7-adde-2d015113976c" containerID="8429056fc870c1bfe0fcecb6a71986b54e0b6005308b1fb61aeb241ff06df78f" exitCode=0 Nov 01 22:48:48.123523 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:48.123165 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" event={"ID":"26bd4957-9025-48d7-adde-2d015113976c","Type":"ContainerDied","Data":"8429056fc870c1bfe0fcecb6a71986b54e0b6005308b1fb61aeb241ff06df78f"} Nov 01 22:48:48.271464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:48.271432 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:49.095039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.094998 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:49.431782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.431756 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:49.512610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.512552 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26bd4957-9025-48d7-adde-2d015113976c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bd4957-9025-48d7-adde-2d015113976c-maas-api-xks-ca\") pod \"26bd4957-9025-48d7-adde-2d015113976c\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " Nov 01 22:48:49.512818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.512776 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26bd4957-9025-48d7-adde-2d015113976c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bd4957-9025-48d7-adde-2d015113976c-maas-api-openshift-ca\") pod \"26bd4957-9025-48d7-adde-2d015113976c\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " Nov 01 22:48:49.513204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.513174 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26bd4957-9025-48d7-adde-2d015113976c-maas-api-openshift-ca" pod "26bd4957-9025-48d7-adde-2d015113976c" (UID: "26bd4957-9025-48d7-adde-2d015113976c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:48:49.513478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.513455 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26bd4957-9025-48d7-adde-2d015113976c-kube-api-access-kqbs7\" (UniqueName: \"kubernetes.io/projected/26bd4957-9025-48d7-adde-2d015113976c-kube-api-access-kqbs7\") pod \"26bd4957-9025-48d7-adde-2d015113976c\" (UID: \"26bd4957-9025-48d7-adde-2d015113976c\") " Nov 01 22:48:49.514745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.514720 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26bd4957-9025-48d7-adde-2d015113976c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:49.515124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.515096 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26bd4957-9025-48d7-adde-2d015113976c-maas-api-xks-ca" pod "26bd4957-9025-48d7-adde-2d015113976c" (UID: "26bd4957-9025-48d7-adde-2d015113976c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:48:49.518624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.516749 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26bd4957-9025-48d7-adde-2d015113976c-kube-api-access-kqbs7" pod "26bd4957-9025-48d7-adde-2d015113976c" (UID: "26bd4957-9025-48d7-adde-2d015113976c"). InnerVolumeSpecName "kube-api-access-kqbs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:48:49.616327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.616276 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26bd4957-9025-48d7-adde-2d015113976c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:49.616327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.616321 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqbs7\" (UniqueName: \"kubernetes.io/projected/26bd4957-9025-48d7-adde-2d015113976c-kube-api-access-kqbs7\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:48:49.975073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:49.975034 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:50.145829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.145198 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" Nov 01 22:48:50.145829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.145202 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr" event={"ID":"26bd4957-9025-48d7-adde-2d015113976c","Type":"ContainerDied","Data":"284a64a7a5221a726807c18bc35649b50a3ffd823bf798dee62809e80e099912"} Nov 01 22:48:50.145829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.145345 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284a64a7a5221a726807c18bc35649b50a3ffd823bf798dee62809e80e099912" Nov 01 22:48:50.657416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.657376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:50.760609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.760558 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv"] Nov 01 22:48:50.762207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.762171 4981 cpu_manager.go:414] "containerMap: removing container" podUID="26bd4957-9025-48d7-adde-2d015113976c" containerName="revoke-keys" Nov 01 22:48:50.762207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.762195 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26bd4957-9025-48d7-adde-2d015113976c" containerName="revoke-keys" Nov 01 22:48:50.762514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.762489 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="26bd4957-9025-48d7-adde-2d015113976c" containerName="revoke-keys" Nov 01 22:48:50.767810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.767782 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.778605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.775915 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv"] Nov 01 22:48:50.853311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.853266 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.853519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.853493 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.853627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.853590 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.854058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.853863 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.854058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.853952 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.854058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.853980 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.854905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.854118 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5w5\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-kube-api-access-5n5w5\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.854905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.854175 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.856206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.856181 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.957563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.957519 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.957812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.957607 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.957812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.957653 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.957812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.957712 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.957863 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.957932 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.957955 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.958015 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.958055 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5w5\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-kube-api-access-5n5w5\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.958098 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.958241 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.958294 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958587 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.958551 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.958890 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.958868 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.960685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.960662 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.960765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.960731 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.967648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.967552 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:50.967773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:50.967755 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5w5\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-kube-api-access-5n5w5\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:51.126694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.126661 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:51.156100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.156070 4981 generic.go:356] "Generic (PLEG): container finished" podID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerID="3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7" exitCode=0 Nov 01 22:48:51.156269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.156144 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" event={"ID":"9efbf218-9374-41a1-bcc3-00b296a7b05d","Type":"ContainerDied","Data":"3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7"} Nov 01 22:48:51.156709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.156693 4981 scope.go:122] "RemoveContainer" containerID="3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7" Nov 01 22:48:51.158920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.158884 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:51.159033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.158962 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:51.159033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.159025 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:51.425836 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:48:51.425799 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6f6927_7cd2_44d1_bc7f_b2bf471a83fa.slice/crio-66609db549b344480a868becacddd693ad62bbf65ba0454655ca13f2dac61175 WatchSource:0}: Error finding container 66609db549b344480a868becacddd693ad62bbf65ba0454655ca13f2dac61175: Status 404 returned error can't find the container with id 66609db549b344480a868becacddd693ad62bbf65ba0454655ca13f2dac61175 Nov 01 22:48:51.426366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.426340 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv"] Nov 01 22:48:51.429008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.428977 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:51.429130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.429044 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:51.429130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.429073 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:48:51.481701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:51.481656 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:52.167241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.167202 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" event={"ID":"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa","Type":"ContainerStarted","Data":"24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8"} Nov 01 22:48:52.167735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.167274 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" event={"ID":"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa","Type":"ContainerStarted","Data":"66609db549b344480a868becacddd693ad62bbf65ba0454655ca13f2dac61175"} Nov 01 22:48:52.169860 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.169824 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" event={"ID":"9efbf218-9374-41a1-bcc3-00b296a7b05d","Type":"ContainerStarted","Data":"a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31"} Nov 01 22:48:52.207938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.207864 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podStartSLOduration=2.207844429 podStartE2EDuration="2.207844429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:52.206844371 +0000 UTC m=+1497.634722304" watchObservedRunningTime="2026-11-01 22:48:52.207844429 +0000 UTC m=+1497.635722362" Nov 01 22:48:52.254254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.254222 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:48:52.255404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.255378 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:48:52.255531 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.255423 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:48:52.354740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:52.354704 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:53.127502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:53.127133 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:53.129136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:53.129110 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 22:48:53.129283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:53.129183 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 22:48:53.207599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:53.203964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:53.256765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:53.255431 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:48:53.256765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:53.255491 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:48:54.008688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:54.008654 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:54.132603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:54.131762 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 22:48:54.132603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:54.131827 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 22:48:54.235709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:54.235611 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:48:54.254149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:54.254119 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:48:54.254315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:54.254183 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:48:54.953149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:54.953119 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:55.143072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:55.143034 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 22:48:55.143357 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:55.143329 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 22:48:55.254168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:55.254087 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:48:55.254530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:55.254152 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:48:55.781599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:55.781196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:56.128651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:56.127993 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 22:48:56.128651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:56.128066 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 22:48:56.235615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:56.234897 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:56.256126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:56.254992 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:48:56.256126 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:56.255054 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:48:56.400687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:56.389491 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:56.763605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:56.759696 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:57.125146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:57.125051 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:48:57.125748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:57.125725 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:48:57.127095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:57.127071 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 22:48:57.127292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:57.127268 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 22:48:57.254120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:57.254087 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:48:57.254303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:57.254162 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:48:57.506128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:57.506081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:58.111250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:58.111168 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:58.130468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:58.130023 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 22:48:58.130468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:58.130085 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 22:48:58.254299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:58.254260 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Nov 01 22:48:58.254483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:58.254331 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Nov 01 22:48:58.826615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:58.824697 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:48:59.132176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:59.132105 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:59.132550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:59.132535 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:59.133274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:59.133253 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:48:59.259401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:59.259371 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:48:59.260548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:59.260528 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:48:59.932233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:59.932198 4981 scope.go:122] "RemoveContainer" containerID="a10b9c8dbef0e49e9078e07395a3ed392861ca22fd5d014ed410e5b4f26436d7" Nov 01 22:48:59.954912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:48:59.954881 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:00.089437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:00.089397 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:00.976905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:00.976874 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:01.986802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:01.986771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:02.350452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:02.350345 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:02.459539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:02.459504 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:03.036411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.036377 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:03.575371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.575335 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:03.660283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.660247 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x"] Nov 01 22:49:03.667304 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.667280 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:03.679560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.679526 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x"] Nov 01 22:49:03.683217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.683192 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-021714\"" Nov 01 22:49:03.772522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.772481 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88lpt\" (UniqueName: \"kubernetes.io/projected/c3c9e058-877f-41e0-8e59-ac5b17f987fc-kube-api-access-88lpt\") pod \"payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:03.773145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.773115 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c9e058-877f-41e0-8e59-ac5b17f987fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:03.978668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:03.879763 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88lpt\" (UniqueName: \"kubernetes.io/projected/c3c9e058-877f-41e0-8e59-ac5b17f987fc-kube-api-access-88lpt\") pod \"payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:04.019068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:04.011191 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:04.038790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:04.038758 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c9e058-877f-41e0-8e59-ac5b17f987fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:04.068167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:04.068131 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c9e058-877f-41e0-8e59-ac5b17f987fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:04.099639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:04.094312 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88lpt\" (UniqueName: \"kubernetes.io/projected/c3c9e058-877f-41e0-8e59-ac5b17f987fc-kube-api-access-88lpt\") pod \"payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:04.287242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:04.287155 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:04.479397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:04.479361 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:04.595670 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:49:04.589796 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c9e058_877f_41e0_8e59_ac5b17f987fc.slice/crio-f179e7de7d5778b6c16ef65da37fc5398c61ee842bf6da91c20235324a8f3149 WatchSource:0}: Error finding container f179e7de7d5778b6c16ef65da37fc5398c61ee842bf6da91c20235324a8f3149: Status 404 returned error can't find the container with id f179e7de7d5778b6c16ef65da37fc5398c61ee842bf6da91c20235324a8f3149 Nov 01 22:49:04.598024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:04.597990 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x"] Nov 01 22:49:05.030976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.030940 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd"] Nov 01 22:49:05.036068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.036026 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183e4b75-snrdd"] Nov 01 22:49:05.176938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.176908 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:05.384659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.376844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:05.670654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.668643 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:05.688597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.685651 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7"] Nov 01 22:49:05.690968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.690162 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" event={"ID":"c3c9e058-877f-41e0-8e59-ac5b17f987fc","Type":"ContainerStarted","Data":"dd379040ccef9eb0d06f9cfa63016765be4badb5b8dcb60f884866172b9d26ff"} Nov 01 22:49:05.690968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.690232 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" event={"ID":"c3c9e058-877f-41e0-8e59-ac5b17f987fc","Type":"ContainerStarted","Data":"f179e7de7d5778b6c16ef65da37fc5398c61ee842bf6da91c20235324a8f3149"} Nov 01 22:49:05.691647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.691566 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:05.695785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.695757 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c0d076-dtwm7"] Nov 01 22:49:05.717647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.717585 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" podStartSLOduration=2.717557374 podStartE2EDuration="2.717557374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:05.713471059 +0000 UTC m=+1511.141349082" watchObservedRunningTime="2026-11-01 22:49:05.717557374 +0000 UTC m=+1511.145435331" Nov 01 22:49:05.780535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:05.780502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:06.188311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:06.188261 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:06.743030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:06.742985 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e8f96b-4bf9-4977-a86d-77aa8ba31f86" path="/var/lib/kubelet/pods/77e8f96b-4bf9-4977-a86d-77aa8ba31f86/volumes" Nov 01 22:49:06.743651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:06.743630 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b" path="/var/lib/kubelet/pods/f98e2cd5-dbb7-4fdf-ab08-7ed8d5178f3b/volumes" Nov 01 22:49:06.777024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:06.776959 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:06.880673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:06.880634 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:07.016659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:07.005637 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:07.798099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:07.798056 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:08.003923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:08.003873 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:08.559680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:08.559631 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk"] Nov 01 22:49:08.577283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:08.577247 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b34688-552pk"] Nov 01 22:49:08.583901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:08.583863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:08.696517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:08.696483 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:08.738773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:08.738736 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57ef9d0-23ad-4885-87c7-9fe034425a10" path="/var/lib/kubelet/pods/e57ef9d0-23ad-4885-87c7-9fe034425a10/volumes" Nov 01 22:49:09.028771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:09.028734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:09.631731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:09.631697 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:10.506060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:10.506021 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:11.379174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:11.379115 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:12.730207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:12.730172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:13.137160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:13.137085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:13.267547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:13.267511 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:13.569606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:13.569544 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg"] Nov 01 22:49:13.569797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:13.569769 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" containerID="cri-o://69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a" gracePeriod=30 Nov 01 22:49:13.749610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:13.749558 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:14.077123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:14.077093 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:14.194903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:14.194863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:14.379766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:14.379677 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:14.527932 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:14.527900 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:49:14.759473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:14.759437 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:14.963484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:14.963450 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:15.212672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:15.193110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:15.328480 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:15.328439 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:16.180414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:16.180374 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:16.497936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:16.497846 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:16.849129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:16.849043 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl"] Nov 01 22:49:16.849320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:16.849277 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" containerID="cri-o://d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8" gracePeriod=30 Nov 01 22:49:16.881650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:16.881613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:17.337244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.337206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:17.380813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.380762 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86"] Nov 01 22:49:17.386898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.386872 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.402411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.402380 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:49:17.402606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.402470 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:49:17.402682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.402642 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:49:17.402848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.402819 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:49:17.409312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.409286 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86"] Nov 01 22:49:17.507524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.507483 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgbh\" (UniqueName: \"kubernetes.io/projected/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-kube-api-access-bmgbh\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.507742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.507530 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.507742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.507673 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.609268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.609176 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgbh\" (UniqueName: \"kubernetes.io/projected/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-kube-api-access-bmgbh\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.609268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.609222 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.609560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.609274 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.610356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.610115 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.614033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.614004 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.620121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.620089 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgbh\" (UniqueName: \"kubernetes.io/projected/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-kube-api-access-bmgbh\") pod \"maas-api-revoke-keys-e2e-api-b-e36490-2mz86\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.709810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.706564 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:17.714469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.714445 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 22:49:17.982606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:17.981800 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:18.038880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:18.038633 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86"] Nov 01 22:49:18.039336 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:49:18.039302 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd314c4_7cba_4ac9_9bc7_5e80de063f8b.slice/crio-53a8ce5d4d0951ec5a091af33ce5f53974e88eb9aa27c2961cf8667ca933fd3e WatchSource:0}: Error finding container 53a8ce5d4d0951ec5a091af33ce5f53974e88eb9aa27c2961cf8667ca933fd3e: Status 404 returned error can't find the container with id 53a8ce5d4d0951ec5a091af33ce5f53974e88eb9aa27c2961cf8667ca933fd3e Nov 01 22:49:18.104270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:18.104235 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" event={"ID":"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b","Type":"ContainerStarted","Data":"53a8ce5d4d0951ec5a091af33ce5f53974e88eb9aa27c2961cf8667ca933fd3e"} Nov 01 22:49:18.204178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:18.204138 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:18.715365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:18.715326 4981 patch_prober.go:29] interesting pod/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:49:18.715810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:18.715381 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:49:18.876318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:18.876258 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:18.890187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:18.890161 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:49:19.011396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.011321 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:19.066502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066467 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-certs\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.066710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066516 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istiod-ca-cert\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.066756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066726 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-kube-api-access-crmgj\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-kube-api-access-crmgj\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.066797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066773 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-credential-socket\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.066856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066801 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-podinfo\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.066856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066806 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-certs" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:19.066856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066830 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-envoy\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.067019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066918 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-socket\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.067019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066945 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-token\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-token\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.067132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.066993 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istiod-ca-cert" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:19.067132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.067100 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-data\") pod \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\" (UID: \"8f2dd39f-6f0f-4481-b36d-6d059a93958c\") " Nov 01 22:49:19.067295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.067272 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-socket" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:19.067460 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.067437 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-credential-socket" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:19.067616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.067565 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-data" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:19.068069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.068044 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.068149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.068076 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.068149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.068092 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.068149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.068106 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.068149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.068120 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.069638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.069618 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-podinfo" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:19.069719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.069641 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-token" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:19.069719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.069652 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-envoy" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:19.069719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.069676 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-kube-api-access-crmgj" pod "8f2dd39f-6f0f-4481-b36d-6d059a93958c" (UID: "8f2dd39f-6f0f-4481-b36d-6d059a93958c"). InnerVolumeSpecName "kube-api-access-crmgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:19.117223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.117184 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" event={"ID":"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b","Type":"ContainerStarted","Data":"46435f6c797d0f90b4c60d8cc6b1ba383402700e57e4193e8e03549ed4dea02a"} Nov 01 22:49:19.119658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.119632 4981 generic.go:356] "Generic (PLEG): container finished" podID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerID="69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a" exitCode=0 Nov 01 22:49:19.119794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.119677 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" Nov 01 22:49:19.119794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.119711 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" event={"ID":"8f2dd39f-6f0f-4481-b36d-6d059a93958c","Type":"ContainerDied","Data":"69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a"} Nov 01 22:49:19.119794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.119758 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg" event={"ID":"8f2dd39f-6f0f-4481-b36d-6d059a93958c","Type":"ContainerDied","Data":"21fd70c0061ceee0912536ed779b7e043653e14f4ca0cf7d84c41ed09b18fe19"} Nov 01 22:49:19.119794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.119783 4981 scope.go:122] "RemoveContainer" containerID="69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a" Nov 01 22:49:19.137748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.137726 4981 scope.go:122] "RemoveContainer" containerID="cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824" Nov 01 22:49:19.145994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.145943 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" podStartSLOduration=2.145926496 podStartE2EDuration="2.145926496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:19.141503912 +0000 UTC m=+1524.569381844" watchObservedRunningTime="2026-11-01 22:49:19.145926496 +0000 UTC m=+1524.573804428" Nov 01 22:49:19.150818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.150799 4981 scope.go:122] "RemoveContainer" containerID="69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a" Nov 01 22:49:19.151149 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:19.151127 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a\": container with ID starting with 69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a not found: ID does not exist" containerID="69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a" Nov 01 22:49:19.151254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.151154 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a"} err="failed to get container status \"69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a\": rpc error: code = NotFound desc = could not find container \"69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a\": container with ID starting with 69399c596d4f97ebae3d86bd6e6d18820f701d3980ff2454e951bc298906b74a not found: ID does not exist" Nov 01 22:49:19.151254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.151171 4981 scope.go:122] "RemoveContainer" containerID="cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824" Nov 01 22:49:19.151443 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:19.151427 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824\": container with ID starting with cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824 not found: ID does not exist" containerID="cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824" Nov 01 22:49:19.151493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.151449 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824"} err="failed to get container status \"cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824\": rpc error: code = NotFound desc = could not find container \"cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824\": container with ID starting with cc075ab37a2a88fcc67e64d51b636f31872bf340b60c1ce5832706e367342824 not found: ID does not exist" Nov 01 22:49:19.175757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.175728 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg"] Nov 01 22:49:19.176344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.176310 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.176465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.176348 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.176465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.176363 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.176465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.176377 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmgj\" (UniqueName: \"kubernetes.io/projected/8f2dd39f-6f0f-4481-b36d-6d059a93958c-kube-api-access-crmgj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:19.191340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.191303 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c0d076-openshift-default-7bff86d6c9-jsrgg"] Nov 01 22:49:19.522003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:19.521954 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:20.093450 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.093401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:20.138851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.138810 4981 generic.go:356] "Generic (PLEG): container finished" podID="8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" containerID="46435f6c797d0f90b4c60d8cc6b1ba383402700e57e4193e8e03549ed4dea02a" exitCode=0 Nov 01 22:49:20.139013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.138934 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" event={"ID":"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b","Type":"ContainerDied","Data":"46435f6c797d0f90b4c60d8cc6b1ba383402700e57e4193e8e03549ed4dea02a"} Nov 01 22:49:20.232605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.232388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:20.623326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.623294 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:20.655282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.655252 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft"] Nov 01 22:49:20.656204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.656187 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" Nov 01 22:49:20.656291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.656206 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" Nov 01 22:49:20.656291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.656238 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" Nov 01 22:49:20.656291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.656243 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" Nov 01 22:49:20.656428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.656418 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" Nov 01 22:49:20.660803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.660785 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:20.671332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.671309 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgmt\" (UniqueName: \"kubernetes.io/projected/369ead52-8316-4afd-9b8f-ab992cd76b7d-kube-api-access-8fgmt\") pod \"payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:20.671834 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.671779 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369ead52-8316-4afd-9b8f-ab992cd76b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:20.672123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.672102 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2f675d\"" Nov 01 22:49:20.672244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.672226 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2f675d-dockercfg-nnrjj\"" Nov 01 22:49:20.675056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.675034 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft"] Nov 01 22:49:20.715426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.715394 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb"] Nov 01 22:49:20.716775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.716753 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" containerName="istio-proxy" Nov 01 22:49:20.720840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.720819 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:20.826600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.821009 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgmt\" (UniqueName: \"kubernetes.io/projected/369ead52-8316-4afd-9b8f-ab992cd76b7d-kube-api-access-8fgmt\") pod \"payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:20.826600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.821415 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369ead52-8316-4afd-9b8f-ab992cd76b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:20.886003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.880408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:20.886003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.880677 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369ead52-8316-4afd-9b8f-ab992cd76b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:21.059599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:20.986824 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgmt\" (UniqueName: \"kubernetes.io/projected/369ead52-8316-4afd-9b8f-ab992cd76b7d-kube-api-access-8fgmt\") pod \"payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:21.059599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.001178 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e21a98-7a1e-45d7-a97f-8344eefe7843-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:21.059599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.001382 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxh9\" (UniqueName: \"kubernetes.io/projected/e1e21a98-7a1e-45d7-a97f-8344eefe7843-kube-api-access-nnxh9\") pod \"payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:21.059599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.003782 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:21.059599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.033967 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:21.266474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.266003 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:21.275347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.273414 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e21a98-7a1e-45d7-a97f-8344eefe7843-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:21.275347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.273631 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxh9\" (UniqueName: \"kubernetes.io/projected/e1e21a98-7a1e-45d7-a97f-8344eefe7843-kube-api-access-nnxh9\") pod \"payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:21.275473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.275433 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e21a98-7a1e-45d7-a97f-8344eefe7843-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:21.300700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.291673 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2dd39f-6f0f-4481-b36d-6d059a93958c" path="/var/lib/kubelet/pods/8f2dd39f-6f0f-4481-b36d-6d059a93958c/volumes" Nov 01 22:49:21.300700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.292128 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb"] Nov 01 22:49:21.301641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.301621 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxh9\" (UniqueName: \"kubernetes.io/projected/e1e21a98-7a1e-45d7-a97f-8344eefe7843-kube-api-access-nnxh9\") pod \"payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:21.396634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.395494 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:21.598383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.588701 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft"] Nov 01 22:49:21.636213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.633910 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:21.715236 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.715196 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-xks-ca\") pod \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " Nov 01 22:49:21.715459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.715293 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-kube-api-access-bmgbh\" (UniqueName: \"kubernetes.io/projected/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-kube-api-access-bmgbh\") pod \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " Nov 01 22:49:21.715558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.715519 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-openshift-ca\") pod \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\" (UID: \"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b\") " Nov 01 22:49:21.717281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.717251 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-openshift-ca" pod "8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" (UID: "8bd314c4-7cba-4ac9-9bc7-5e80de063f8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:21.721134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.721106 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-xks-ca" pod "8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" (UID: "8bd314c4-7cba-4ac9-9bc7-5e80de063f8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:49:21.722929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.722904 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-kube-api-access-bmgbh" pod "8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" (UID: "8bd314c4-7cba-4ac9-9bc7-5e80de063f8b"). InnerVolumeSpecName "kube-api-access-bmgbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:21.819308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.819253 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:21.819308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.819285 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:21.819308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.819300 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmgbh\" (UniqueName: \"kubernetes.io/projected/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b-kube-api-access-bmgbh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:21.861111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:21.861073 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb"] Nov 01 22:49:21.861359 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:49:21.861332 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e21a98_7a1e_45d7_a97f_8344eefe7843.slice/crio-5d5d1eb7c72da78345b92547d87b197f4b4cec0f1e1b818edfea8a274e80b9ce WatchSource:0}: Error finding container 5d5d1eb7c72da78345b92547d87b197f4b4cec0f1e1b818edfea8a274e80b9ce: Status 404 returned error can't find the container with id 5d5d1eb7c72da78345b92547d87b197f4b4cec0f1e1b818edfea8a274e80b9ce Nov 01 22:49:22.154056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.154027 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:49:22.202961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.202874 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:22.226380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226348 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-credential-socket\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.226614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226392 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-envoy\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.226614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226502 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-certs\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.226614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226547 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-socket\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.226799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226634 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-kube-api-access-79jn6\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-kube-api-access-79jn6\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.226799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226675 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-credential-socket" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:22.226799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226731 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istiod-ca-cert\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.226982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226856 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-certs" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:22.226982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.226914 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-socket" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:22.227102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.227063 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-data" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:22.227164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.227099 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istiod-ca-cert" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:22.227164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.227137 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-data\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.227308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.227291 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-podinfo\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.227489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.227474 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-token\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-token\") pod \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\" (UID: \"e49ed163-c875-4004-9f6e-3a5059d8ccf1\") " Nov 01 22:49:22.228391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.228361 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.228513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.228393 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.228513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.228408 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.228513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.228423 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.228513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.228439 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.229087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.229058 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-envoy" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:22.229179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.229125 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-kube-api-access-79jn6" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "kube-api-access-79jn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:22.229433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.229418 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-podinfo" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:22.229990 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.229969 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-token" pod "e49ed163-c875-4004-9f6e-3a5059d8ccf1" (UID: "e49ed163-c875-4004-9f6e-3a5059d8ccf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:22.290740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.290610 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" event={"ID":"369ead52-8316-4afd-9b8f-ab992cd76b7d","Type":"ContainerStarted","Data":"aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65"} Nov 01 22:49:22.290740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.290676 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:22.290740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.290699 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" event={"ID":"369ead52-8316-4afd-9b8f-ab992cd76b7d","Type":"ContainerStarted","Data":"cdaf679ddae454621f47f02d9d33a7a5993ec57e675fdd772baf86380aac1f8c"} Nov 01 22:49:22.315731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.315681 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" event={"ID":"e1e21a98-7a1e-45d7-a97f-8344eefe7843","Type":"ContainerStarted","Data":"eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546"} Nov 01 22:49:22.316023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.315997 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" event={"ID":"e1e21a98-7a1e-45d7-a97f-8344eefe7843","Type":"ContainerStarted","Data":"5d5d1eb7c72da78345b92547d87b197f4b4cec0f1e1b818edfea8a274e80b9ce"} Nov 01 22:49:22.316970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.316947 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:22.318614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.318591 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" event={"ID":"8bd314c4-7cba-4ac9-9bc7-5e80de063f8b","Type":"ContainerDied","Data":"53a8ce5d4d0951ec5a091af33ce5f53974e88eb9aa27c2961cf8667ca933fd3e"} Nov 01 22:49:22.319264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.319157 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53a8ce5d4d0951ec5a091af33ce5f53974e88eb9aa27c2961cf8667ca933fd3e" Nov 01 22:49:22.319427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.318638 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86" Nov 01 22:49:22.322751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.322728 4981 generic.go:356] "Generic (PLEG): container finished" podID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerID="d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8" exitCode=0 Nov 01 22:49:22.322854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.322785 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" Nov 01 22:49:22.322854 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.322809 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" event={"ID":"e49ed163-c875-4004-9f6e-3a5059d8ccf1","Type":"ContainerDied","Data":"d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8"} Nov 01 22:49:22.322944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.322869 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl" event={"ID":"e49ed163-c875-4004-9f6e-3a5059d8ccf1","Type":"ContainerDied","Data":"d42fb4be0c98bd2584b533a47abf13939087c369b1ea560f894b717cd5cfb839"} Nov 01 22:49:22.322944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.322899 4981 scope.go:122] "RemoveContainer" containerID="d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8" Nov 01 22:49:22.332244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.332008 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" podStartSLOduration=2.331990218 podStartE2EDuration="2.331990218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:22.324923926 +0000 UTC m=+1527.752801871" watchObservedRunningTime="2026-11-01 22:49:22.331990218 +0000 UTC m=+1527.759868150" Nov 01 22:49:22.332719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.332696 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.332874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.332859 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.332969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.332955 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79jn6\" (UniqueName: \"kubernetes.io/projected/e49ed163-c875-4004-9f6e-3a5059d8ccf1-kube-api-access-79jn6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.333061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.333049 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49ed163-c875-4004-9f6e-3a5059d8ccf1-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:22.347571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.347517 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" podStartSLOduration=2.347497857 podStartE2EDuration="2.347497857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:22.342430693 +0000 UTC m=+1527.770308625" watchObservedRunningTime="2026-11-01 22:49:22.347497857 +0000 UTC m=+1527.775375788" Nov 01 22:49:22.359687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.359646 4981 scope.go:122] "RemoveContainer" containerID="8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928" Nov 01 22:49:22.371332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.371298 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl"] Nov 01 22:49:22.375911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.375888 4981 scope.go:122] "RemoveContainer" containerID="d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8" Nov 01 22:49:22.376227 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:22.376208 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8\": container with ID starting with d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8 not found: ID does not exist" containerID="d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8" Nov 01 22:49:22.376294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.376238 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8"} err="failed to get container status \"d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8\": rpc error: code = NotFound desc = could not find container \"d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8\": container with ID starting with d5e97342893a64d0534b188160f9f04750b88724d7ee1824c02968dc79c609c8 not found: ID does not exist" Nov 01 22:49:22.376294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.376256 4981 scope.go:122] "RemoveContainer" containerID="8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928" Nov 01 22:49:22.376539 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:22.376523 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928\": container with ID starting with 8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928 not found: ID does not exist" containerID="8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928" Nov 01 22:49:22.376637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.376544 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928"} err="failed to get container status \"8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928\": rpc error: code = NotFound desc = could not find container \"8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928\": container with ID starting with 8741ad213954fb67cf537862c0c3b326806cfc55d08326964cfca479099c6928 not found: ID does not exist" Nov 01 22:49:22.377525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.377502 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b34688-openshift-default-6b79c696ff-cbbgl"] Nov 01 22:49:22.737775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:22.737733 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" path="/var/lib/kubelet/pods/e49ed163-c875-4004-9f6e-3a5059d8ccf1/volumes" Nov 01 22:49:23.296885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:23.296851 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:24.381236 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:24.380186 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:25.075685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:25.075652 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:25.765718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:25.765681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:26.470167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:26.470135 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:27.292239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:27.292201 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:27.516863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:27.516820 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d"] Nov 01 22:49:27.535193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:27.535159 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr"] Nov 01 22:49:27.596320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:27.596231 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:27.695249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:27.695214 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:28.536682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:28.535951 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.165610 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4"] Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.167726 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" containerName="revoke-keys" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.167749 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" containerName="revoke-keys" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.167772 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.167779 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.167821 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.167831 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.168081 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" containerName="revoke-keys" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.168101 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.168118 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49ed163-c875-4004-9f6e-3a5059d8ccf1" containerName="istio-proxy" Nov 01 22:49:29.178259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.175178 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.189677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.189630 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 22:49:29.191650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.190998 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 22:49:29.197610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.197566 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4"] Nov 01 22:49:29.199007 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.198975 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd491ae-556d-4ce8-a5ae-05226714e9ef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.199153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.199132 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-metrics-certs\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.199225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.199186 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t758n\" (UniqueName: \"kubernetes.io/projected/6fd491ae-556d-4ce8-a5ae-05226714e9ef-kube-api-access-t758n\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.199290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.199241 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.211334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.211302 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:29.214443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.214216 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-021714-serving-cert\"" Nov 01 22:49:29.214443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.214227 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 22:49:29.214443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.214386 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 22:49:29.312399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.312260 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-metrics-certs\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.312399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.312350 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t758n\" (UniqueName: \"kubernetes.io/projected/6fd491ae-556d-4ce8-a5ae-05226714e9ef-kube-api-access-t758n\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.312399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.312392 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.312871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.312850 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd491ae-556d-4ce8-a5ae-05226714e9ef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.314136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.313644 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd491ae-556d-4ce8-a5ae-05226714e9ef-openshift-service-ca\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.316313 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:29.314757 4981 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-serving-cert: secret "maas-api-e2e-inf-a-021714-serving-cert" not found Nov 01 22:49:29.316313 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:29.314855 4981 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls podName:6fd491ae-556d-4ce8-a5ae-05226714e9ef nodeName:}" failed. No retries permitted until 2026-11-01 22:49:29.814833704 +0000 UTC m=+1535.242711633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls") pod "maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" (UID: "6fd491ae-556d-4ce8-a5ae-05226714e9ef") : secret "maas-api-e2e-inf-a-021714-serving-cert" not found Nov 01 22:49:29.319187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.319147 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-metrics-certs\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.320096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.320056 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:29.326313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.326250 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t758n\" (UniqueName: \"kubernetes.io/projected/6fd491ae-556d-4ce8-a5ae-05226714e9ef-kube-api-access-t758n\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.528938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.528859 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:29.821598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.821493 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:29.823971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:29.823947 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls\") pod \"maas-api-e2e-inf-a-021714-85885b87f5-xvxf4\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:30.106118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:30.105992 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:30.434395 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:49:30.433899 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd491ae_556d_4ce8_a5ae_05226714e9ef.slice/crio-9c392d0dd6353dcbcce8f021938d48e1c30932f28c3ce6b542871a21ea3e797b WatchSource:0}: Error finding container 9c392d0dd6353dcbcce8f021938d48e1c30932f28c3ce6b542871a21ea3e797b: Status 404 returned error can't find the container with id 9c392d0dd6353dcbcce8f021938d48e1c30932f28c3ce6b542871a21ea3e797b Nov 01 22:49:30.435437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:30.435411 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:49:30.436648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:30.436507 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:30.440075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:30.440049 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4"] Nov 01 22:49:30.462010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:30.461971 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" event={"ID":"6fd491ae-556d-4ce8-a5ae-05226714e9ef","Type":"ContainerStarted","Data":"9c392d0dd6353dcbcce8f021938d48e1c30932f28c3ce6b542871a21ea3e797b"} Nov 01 22:49:31.163653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:31.163613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:31.969334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:31.969294 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:32.485410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:32.485373 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" event={"ID":"6fd491ae-556d-4ce8-a5ae-05226714e9ef","Type":"ContainerStarted","Data":"2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813"} Nov 01 22:49:32.485925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:32.485454 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:32.508077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:32.508023 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" podStartSLOduration=1.6815950659999999 podStartE2EDuration="3.508008226s" totalImagesPullingTime="1.82641316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:32.506622887 +0000 UTC m=+1537.934500821" watchObservedRunningTime="2026-11-01 22:49:32.508008226 +0000 UTC m=+1537.935886157" Nov 01 22:49:32.517558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:32.517476 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" podUID="9d83862c-2c75-4fd8-8034-cb297feb10ce" containerName="payload-processing" containerID="cri-o://8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c" gracePeriod=25 Nov 01 22:49:32.536454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:32.536416 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" podUID="106ec4c2-48f8-43d6-9dec-64bd302aa012" containerName="payload-pre-processing" containerID="cri-o://22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95" gracePeriod=25 Nov 01 22:49:32.710611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:32.708005 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:32.970836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:32.970802 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:49:33.017996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.017968 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:49:33.034922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.034884 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d83862c-2c75-4fd8-8034-cb297feb10ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d83862c-2c75-4fd8-8034-cb297feb10ce-plugins-config-volume\") pod \"9d83862c-2c75-4fd8-8034-cb297feb10ce\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " Nov 01 22:49:33.035260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.035238 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106ec4c2-48f8-43d6-9dec-64bd302aa012-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106ec4c2-48f8-43d6-9dec-64bd302aa012-plugins-config-volume\") pod \"106ec4c2-48f8-43d6-9dec-64bd302aa012\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " Nov 01 22:49:33.035381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.035289 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106ec4c2-48f8-43d6-9dec-64bd302aa012-kube-api-access-fzxmm\" (UniqueName: \"kubernetes.io/projected/106ec4c2-48f8-43d6-9dec-64bd302aa012-kube-api-access-fzxmm\") pod \"106ec4c2-48f8-43d6-9dec-64bd302aa012\" (UID: \"106ec4c2-48f8-43d6-9dec-64bd302aa012\") " Nov 01 22:49:33.035381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.035341 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d83862c-2c75-4fd8-8034-cb297feb10ce-plugins-config-volume" pod "9d83862c-2c75-4fd8-8034-cb297feb10ce" (UID: "9d83862c-2c75-4fd8-8034-cb297feb10ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:33.035505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.035408 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d83862c-2c75-4fd8-8034-cb297feb10ce-kube-api-access-xzjc5\" (UniqueName: \"kubernetes.io/projected/9d83862c-2c75-4fd8-8034-cb297feb10ce-kube-api-access-xzjc5\") pod \"9d83862c-2c75-4fd8-8034-cb297feb10ce\" (UID: \"9d83862c-2c75-4fd8-8034-cb297feb10ce\") " Nov 01 22:49:33.037518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.035677 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106ec4c2-48f8-43d6-9dec-64bd302aa012-plugins-config-volume" pod "106ec4c2-48f8-43d6-9dec-64bd302aa012" (UID: "106ec4c2-48f8-43d6-9dec-64bd302aa012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:33.037518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.036434 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/106ec4c2-48f8-43d6-9dec-64bd302aa012-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:33.037518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.036458 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d83862c-2c75-4fd8-8034-cb297feb10ce-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:33.037805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.037744 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106ec4c2-48f8-43d6-9dec-64bd302aa012-kube-api-access-fzxmm" pod "106ec4c2-48f8-43d6-9dec-64bd302aa012" (UID: "106ec4c2-48f8-43d6-9dec-64bd302aa012"). InnerVolumeSpecName "kube-api-access-fzxmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:33.038341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.038313 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d83862c-2c75-4fd8-8034-cb297feb10ce-kube-api-access-xzjc5" pod "9d83862c-2c75-4fd8-8034-cb297feb10ce" (UID: "9d83862c-2c75-4fd8-8034-cb297feb10ce"). InnerVolumeSpecName "kube-api-access-xzjc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:33.138609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.138503 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzxmm\" (UniqueName: \"kubernetes.io/projected/106ec4c2-48f8-43d6-9dec-64bd302aa012-kube-api-access-fzxmm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:33.138609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.138536 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjc5\" (UniqueName: \"kubernetes.io/projected/9d83862c-2c75-4fd8-8034-cb297feb10ce-kube-api-access-xzjc5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:33.360968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.360936 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 22:49:33.497858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.497815 4981 generic.go:356] "Generic (PLEG): container finished" podID="9d83862c-2c75-4fd8-8034-cb297feb10ce" containerID="8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c" exitCode=0 Nov 01 22:49:33.498371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.497866 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" Nov 01 22:49:33.498371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.498041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" event={"ID":"9d83862c-2c75-4fd8-8034-cb297feb10ce","Type":"ContainerDied","Data":"8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c"} Nov 01 22:49:33.498371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.498081 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d" event={"ID":"9d83862c-2c75-4fd8-8034-cb297feb10ce","Type":"ContainerDied","Data":"b37d001068841a8c875d87ccd82a334a10651ccf64cf572941f9ba6496005d6c"} Nov 01 22:49:33.498371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.498102 4981 scope.go:122] "RemoveContainer" containerID="8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c" Nov 01 22:49:33.500077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.500051 4981 generic.go:356] "Generic (PLEG): container finished" podID="106ec4c2-48f8-43d6-9dec-64bd302aa012" containerID="22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95" exitCode=0 Nov 01 22:49:33.500192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.500084 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" Nov 01 22:49:33.500192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.500130 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" event={"ID":"106ec4c2-48f8-43d6-9dec-64bd302aa012","Type":"ContainerDied","Data":"22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95"} Nov 01 22:49:33.500192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.500176 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr" event={"ID":"106ec4c2-48f8-43d6-9dec-64bd302aa012","Type":"ContainerDied","Data":"01bd32f34cf26a442e7856b8c046f0666b8aacf2e8b54c4e79ededb749217e55"} Nov 01 22:49:33.512229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.512207 4981 scope.go:122] "RemoveContainer" containerID="8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c" Nov 01 22:49:33.512541 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:33.512515 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c\": container with ID starting with 8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c not found: ID does not exist" containerID="8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c" Nov 01 22:49:33.512695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.512552 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c"} err="failed to get container status \"8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c\": rpc error: code = NotFound desc = could not find container \"8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c\": container with ID starting with 8d4c5aa365f7f7578f246c50a72ec6c8975a2294977ea5d2e9a951c11b1e195c not found: ID does not exist" Nov 01 22:49:33.512695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.512570 4981 scope.go:122] "RemoveContainer" containerID="22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95" Nov 01 22:49:33.523948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.523929 4981 scope.go:122] "RemoveContainer" containerID="22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95" Nov 01 22:49:33.524249 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:33.524223 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95\": container with ID starting with 22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95 not found: ID does not exist" containerID="22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95" Nov 01 22:49:33.524305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.524258 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95"} err="failed to get container status \"22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95\": rpc error: code = NotFound desc = could not find container \"22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95\": container with ID starting with 22d0585817d6802c97afac97e3830e7a6c8cd3cae63864f27a51594a71f3bd95 not found: ID does not exist" Nov 01 22:49:33.539913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.539889 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d"] Nov 01 22:49:33.554091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.554061 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e36490-55cbccd75f-rmn4d"] Nov 01 22:49:33.572472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.572435 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr"] Nov 01 22:49:33.580562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.580533 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e36490-5fdd775cd7-s6mfr"] Nov 01 22:49:33.600195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:33.600161 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:34.399719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:34.399684 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 22:49:34.831600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:34.822305 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106ec4c2-48f8-43d6-9dec-64bd302aa012" path="/var/lib/kubelet/pods/106ec4c2-48f8-43d6-9dec-64bd302aa012/volumes" Nov 01 22:49:34.831600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:34.823109 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d83862c-2c75-4fd8-8034-cb297feb10ce" path="/var/lib/kubelet/pods/9d83862c-2c75-4fd8-8034-cb297feb10ce/volumes" Nov 01 22:49:35.198035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.197999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:35.832599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.831490 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9"] Nov 01 22:49:35.833101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.833047 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9d83862c-2c75-4fd8-8034-cb297feb10ce" containerName="payload-processing" Nov 01 22:49:35.833101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.833071 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d83862c-2c75-4fd8-8034-cb297feb10ce" containerName="payload-processing" Nov 01 22:49:35.833251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.833182 4981 cpu_manager.go:414] "containerMap: removing container" podUID="106ec4c2-48f8-43d6-9dec-64bd302aa012" containerName="payload-pre-processing" Nov 01 22:49:35.833251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.833191 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106ec4c2-48f8-43d6-9dec-64bd302aa012" containerName="payload-pre-processing" Nov 01 22:49:35.839598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.833415 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="106ec4c2-48f8-43d6-9dec-64bd302aa012" containerName="payload-pre-processing" Nov 01 22:49:35.839598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.833439 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d83862c-2c75-4fd8-8034-cb297feb10ce" containerName="payload-processing" Nov 01 22:49:35.860599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.840202 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.905611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.903648 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9"] Nov 01 22:49:35.975550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.975510 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45pf\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-kube-api-access-f45pf\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.975856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.975828 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.976324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.976302 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.976450 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.976436 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19d413e-0380-4383-968c-d8de43fc8dd9-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.976595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.976564 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.980644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.980619 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.980789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.980774 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.993599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.980933 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:35.993958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:35.993923 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097114 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097178 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19d413e-0380-4383-968c-d8de43fc8dd9-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097219 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097252 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097281 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097345 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097408 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097532 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45pf\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-kube-api-access-f45pf\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.097614 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.142173 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-certs\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.201945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.148318 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-credential-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.247668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.245308 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-token\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.251768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.249879 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-socket\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.251768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.251709 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19d413e-0380-4383-968c-d8de43fc8dd9-istiod-ca-cert\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.253052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.253028 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-data\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.256849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.256810 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-envoy\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.260803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.260750 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-podinfo\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.261152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.261132 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45pf\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-kube-api-access-f45pf\") pod \"e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.368845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.350001 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:36.733322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:36.733281 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:37.377765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.377701 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:49:37.472085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.472038 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:37.548274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.474657 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:37.548274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.484776 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9"] Nov 01 22:49:37.548274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.498440 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:37.608659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.574832 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:37.609163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.609117 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:37.609392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.609372 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:37.837602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.812462 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Nov 01 22:49:37.895659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.891365 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:37.912452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:37.912313 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.030157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.030103 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.034602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.030428 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.042607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.030462 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:49:38.090694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.030491 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.098385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.098183 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:38.178723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.178670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.274607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.273639 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.324602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.303653 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.394606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.362764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:38.394606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.385990 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.394606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.387369 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:49:38.414600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.411038 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" event={"ID":"b19d413e-0380-4383-968c-d8de43fc8dd9","Type":"ContainerStarted","Data":"f10c4bbee8d869c42ba7a0998b26cecf8c1061841d0b418a79f4c54d08617eb4"} Nov 01 22:49:38.475600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.474590 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.475600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.475164 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:49:38.500629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.475878 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.500629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.476318 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:49:38.525602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.518555 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.525602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.519128 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:49:38.700664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.699726 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 22:49:38.900600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.900054 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.929603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.914927 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:38.958661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:38.948775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:39.021602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.017809 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:49:39.062604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.032257 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.065090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.065036 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.126637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.113024 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:39.214152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.184207 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.214152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.185235 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:49:39.214152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.203438 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:49:39.261956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.234381 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.374726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.292251 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.374726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.323212 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:39.375130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.375089 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:49:39.438413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.430681 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:39.908872 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:39.908834 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.147s" Nov 01 22:49:39.975607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.971703 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:49:40.011603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011420 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:49:40.011603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011479 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:49:40.011603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011517 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:49:40.011603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011553 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:49:40.011603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011606 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:49:40.012017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011645 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:49:40.012017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011679 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:49:40.012017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.011717 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:49:40.012285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.536322 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Nov 01 22:49:40.012544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.984820 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.012948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.639369 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:49:40.053469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.520597 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:40.053866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.639408 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:49:40.117912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:39.520670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.117912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.110918 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.118802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.118555 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.142892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.142861 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:49:40.331427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.205237 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:49:40.512740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.205065 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:49:40.586627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.205299 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:49:40.621734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.205399 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:49:40.887745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.204937 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:49:40.891246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.891213 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:49:40.910750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.687003 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:40.911411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.722832 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:40.939232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.939184 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:49:41.016832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:40.996174 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.018520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.014697 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:49:41.086002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.085951 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:49:41.132918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.132867 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.289674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.156361 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.289674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.164081 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:49:41.393612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.136211 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:49:41.496140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.496091 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.569695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.555992 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.569695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.556118 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:49:41.593602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.577124 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 01 22:49:41.610474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.610424 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.887037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.886823 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:41.912270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.910718 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:49:41.947221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:41.934607 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:49:42.037294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.031943 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.037294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.032637 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:49:42.037294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.033849 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:49:42.113604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.111708 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.148707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.140818 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr"] Nov 01 22:49:42.161968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.159518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:42.404254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.403684 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:42.472140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.472095 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.719600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.497659 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:49:42.719600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.497701 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:49:42.728724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.497723 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:49:42.748608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.561349 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr"] Nov 01 22:49:42.749296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.561421 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:42.750248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.682107 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9c89\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-kube-api-access-c9c89\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:42.752510 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.752486 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:42.758388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.758360 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-istio-token\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:42.758676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.758656 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-data\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:42.763627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.698308 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:42.779607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.767967 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0e685d-openshift-default-dockercfg-w4dhv\"" Nov 01 22:49:42.975161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.957998 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" event={"ID":"b19d413e-0380-4383-968c-d8de43fc8dd9","Type":"ContainerStarted","Data":"271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe"} Nov 01 22:49:42.975161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.958093 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:49:42.975161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.958115 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs"] Nov 01 22:49:42.975161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.973160 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:49:42.975161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.973228 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:49:43.024479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:42.988095 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" containerID="cri-o://54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1" gracePeriod=30 Nov 01 22:49:43.041779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.041736 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:49:43.043870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.043846 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:49:43.061815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.055041 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-credential-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.061815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.055112 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-envoy\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.061815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.055213 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a8bacc-1268-463d-b491-b9a705807834-istio-podinfo\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.061815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.055240 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a8bacc-1268-463d-b491-b9a705807834-istiod-ca-cert\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.061815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.055445 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-certs\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.061815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.057075 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:49:43.061815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.061285 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:43.070902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.070870 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:49:43.071715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.070870 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:43.072058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.070915 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:49:43.072329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.070938 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:49:43.163984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.132337 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:43.163984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.157853 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr"] Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.272780 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-credential-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.272862 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-envoy\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.272997 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a8bacc-1268-463d-b491-b9a705807834-istio-podinfo\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.273025 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a8bacc-1268-463d-b491-b9a705807834-istiod-ca-cert\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.273256 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-certs\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.273466 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9c89\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-kube-api-access-c9c89\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.273541 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.273613 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-istio-token\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.275654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.273752 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-data\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.699371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.476517 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:43.699371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.520539 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-credential-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.699371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.520820 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-certs\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.699371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.545313 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a8bacc-1268-463d-b491-b9a705807834-istiod-ca-cert\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.699371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.506176 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-data\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:43.699371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:43.546990 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-socket\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:44.124603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.121868 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:49:44.124603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.122468 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:44.196975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.196918 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:44.199524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.199486 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:49:44.201986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.201952 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:49:44.203068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.197335 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:44.325230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.325146 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr"] Nov 01 22:49:44.451602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.400001 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:49:44.473775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.473717 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:49:44.474163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.474142 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:44.474342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.215745 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:49:44.474718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.209864 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:44.476896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.211516 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-envoy\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:44.477064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.270395 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-istio-token\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:44.477166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.327806 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.480933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.216680 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:49:44.481601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.197513 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 22:49:44.481777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.481755 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 22:49:44.482113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.205649 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:49:44.483266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.390934 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9c89\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-kube-api-access-c9c89\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:44.483442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.415555 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 01 22:49:44.483833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.209364 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:49:44.484497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.473600 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:44.490605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.489591 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 22:49:44.490605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.489659 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 22:49:44.516600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.514508 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:49:44.534072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.533680 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:44.818507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.818458 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/891d2984-c0a2-43c0-9d7a-62e7212486a3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.819021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.818999 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.819239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.819223 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.819503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.819473 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.819896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.819876 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.820266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.820225 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b267g\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-kube-api-access-b267g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.820565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.820547 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.840207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.824683 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a8bacc-1268-463d-b491-b9a705807834-istio-podinfo\") pod \"e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:44.853636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.853600 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:44.873034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.872997 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:44.944966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.944924 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.994016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.948233 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:44.994883 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:44.992634 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:49:44.996112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:44.996092 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b267g\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-kube-api-access-b267g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.017955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.017917 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.018119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.018045 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/891d2984-c0a2-43c0-9d7a-62e7212486a3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.018275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.018254 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.018399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.018331 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.018467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.018435 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.028463 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.028416 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.032175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.030777 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/891d2984-c0a2-43c0-9d7a-62e7212486a3-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.035417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.032847 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.035417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.035191 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.044222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.044189 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:49:45.044376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.044251 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:49:45.044376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.044283 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:49:45.044376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.044311 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:49:45.055596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.054796 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podStartSLOduration=10.054776581 podStartE2EDuration="10.054776581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:44.391837752 +0000 UTC m=+1549.819715681" watchObservedRunningTime="2026-11-01 22:49:45.054776581 +0000 UTC m=+1550.482654515" Nov 01 22:49:45.062643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.062251 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:49:45.074813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.071429 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b267g\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-kube-api-access-b267g\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.121615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.121559 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:49:45.122679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.122653 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:49:45.140687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.123471 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:49:45.165561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.165521 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h9fzr\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.354764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.353979 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 22:49:45.354764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.354035 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 22:49:45.369208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.364853 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr"] Nov 01 22:49:45.370600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.369327 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:45.370600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.369419 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:45.370600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.369467 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:45.451028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.450994 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:45.514669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:45.514639 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:46.043168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.042829 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" event={"ID":"08a8bacc-1268-463d-b491-b9a705807834","Type":"ContainerStarted","Data":"7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e"} Nov 01 22:49:46.043168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.042885 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" event={"ID":"08a8bacc-1268-463d-b491-b9a705807834","Type":"ContainerStarted","Data":"6dcac34cf36629d8bdec6a47a0efa49e08a222aaf03bcbb4921be1044d171fa7"} Nov 01 22:49:46.131297 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:49:46.131248 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891d2984_c0a2_43c0_9d7a_62e7212486a3.slice/crio-4b040d6f17fd404933a372e4a1c2650d4663d0ce5bf14f2ad1978b39daa79149 WatchSource:0}: Error finding container 4b040d6f17fd404933a372e4a1c2650d4663d0ce5bf14f2ad1978b39daa79149: Status 404 returned error can't find the container with id 4b040d6f17fd404933a372e4a1c2650d4663d0ce5bf14f2ad1978b39daa79149 Nov 01 22:49:46.132609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.132259 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr"] Nov 01 22:49:46.135753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.135303 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:46.135753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.135376 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:46.156923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.141091 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podStartSLOduration=7.141074639 podStartE2EDuration="7.141074639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:46.109063126 +0000 UTC m=+1551.536941061" watchObservedRunningTime="2026-11-01 22:49:46.141074639 +0000 UTC m=+1551.568952571" Nov 01 22:49:46.360733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.360679 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 22:49:46.360923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.360740 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 22:49:46.613634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.613534 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:46.881701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.876274 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:46.912643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:46.883454 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:47.075600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.074007 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" event={"ID":"891d2984-c0a2-43c0-9d7a-62e7212486a3","Type":"ContainerStarted","Data":"466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6"} Nov 01 22:49:47.075600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.074075 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:47.075600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.074107 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" event={"ID":"891d2984-c0a2-43c0-9d7a-62e7212486a3","Type":"ContainerStarted","Data":"4b040d6f17fd404933a372e4a1c2650d4663d0ce5bf14f2ad1978b39daa79149"} Nov 01 22:49:47.092741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.087887 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:47.099495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.098973 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:49:47.099495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.099170 4981 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-h9fzr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:49:47.099495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.099213 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:49:47.186819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.177213 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" podStartSLOduration=5.177192764 podStartE2EDuration="5.177192764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:47.16767465 +0000 UTC m=+1552.595552565" watchObservedRunningTime="2026-11-01 22:49:47.177192764 +0000 UTC m=+1552.605070695" Nov 01 22:49:47.401349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.401290 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 01 22:49:47.401552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.401377 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 01 22:49:47.569751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:47.569356 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:48.100465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.100438 4981 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-h9fzr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:49:48.100751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.100725 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:49:48.362997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.362915 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:48.364409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.364320 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:48.365867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.365831 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:49:48.388039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.388008 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:49:48.419033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.418998 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:48.451883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.451850 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istiod-ca-cert\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.452135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.451894 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-data\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.452135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.451970 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-envoy\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.452135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.452110 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-certs\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.452350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.452332 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-socket\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.453879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.452368 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-kube-api-access-qjh5t\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-kube-api-access-qjh5t\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.453879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.453136 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-podinfo\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.453879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.453186 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-credential-socket\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.453879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.453300 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-token\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-token\") pod \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\" (UID: \"3538e3c0-6abd-42ce-9508-06fbf69b6e48\") " Nov 01 22:49:48.456406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.456372 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-credential-socket" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:48.461313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.460449 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-data" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:48.461313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.460462 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-token" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:48.461313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.460650 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istiod-ca-cert" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:49:48.461313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.460785 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-podinfo" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:49:48.461949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.461927 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-envoy" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:48.462664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.462647 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-kube-api-access-qjh5t" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "kube-api-access-qjh5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:49:48.462760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.462686 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-socket" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:48.462760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.462748 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-certs" pod "3538e3c0-6abd-42ce-9508-06fbf69b6e48" (UID: "3538e3c0-6abd-42ce-9508-06fbf69b6e48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559439 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559474 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559489 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559503 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559517 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559529 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559541 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559554 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3538e3c0-6abd-42ce-9508-06fbf69b6e48-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.559567 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjh5t\" (UniqueName: \"kubernetes.io/projected/3538e3c0-6abd-42ce-9508-06fbf69b6e48-kube-api-access-qjh5t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:49:48.577637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:48.576388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:49.071359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.071266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:49.115054 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.115013 4981 generic.go:356] "Generic (PLEG): container finished" podID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerID="54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1" exitCode=0 Nov 01 22:49:49.115232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.115058 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" event={"ID":"3538e3c0-6abd-42ce-9508-06fbf69b6e48","Type":"ContainerDied","Data":"54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1"} Nov 01 22:49:49.115232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.115078 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" Nov 01 22:49:49.115232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.115119 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs" event={"ID":"3538e3c0-6abd-42ce-9508-06fbf69b6e48","Type":"ContainerDied","Data":"1770e77d2fa6cff9516cac3cbde47cd84acc3ace984537a6a04149e3e952e3e6"} Nov 01 22:49:49.115232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.115149 4981 scope.go:122] "RemoveContainer" containerID="54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1" Nov 01 22:49:49.118294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.118268 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:49:49.153043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.149863 4981 scope.go:122] "RemoveContainer" containerID="54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1" Nov 01 22:49:49.153043 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:49:49.150543 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1\": container with ID starting with 54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1 not found: ID does not exist" containerID="54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1" Nov 01 22:49:49.153043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.150625 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1"} err="failed to get container status \"54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1\": rpc error: code = NotFound desc = could not find container \"54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1\": container with ID starting with 54816c34a5fe5b21c4c30b1399f18ae6a0142780500ec6ee400b774def1a1be1 not found: ID does not exist" Nov 01 22:49:49.193597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.192744 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs"] Nov 01 22:49:49.248080 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.248042 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-pzbqs"] Nov 01 22:49:49.730240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.730185 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:49.859229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:49.859193 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:50.468033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:50.467989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:50.740715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:50.740622 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" path="/var/lib/kubelet/pods/3538e3c0-6abd-42ce-9508-06fbf69b6e48/volumes" Nov 01 22:49:51.200250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:51.200211 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:51.874818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:51.874765 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:49:51.875450 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:51.874867 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:49:52.069688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:52.069655 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:52.791505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:52.791469 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:53.409155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.409119 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99"] Nov 01 22:49:53.410131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.410075 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" Nov 01 22:49:53.410131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.410090 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" Nov 01 22:49:53.410283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.410272 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3538e3c0-6abd-42ce-9508-06fbf69b6e48" containerName="istio-proxy" Nov 01 22:49:53.415129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.415102 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.432556 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.432520 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99"] Nov 01 22:49:53.480274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480239 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.480498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480465 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.480604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480570 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.480770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480623 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.480770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480700 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63056725-77e3-4c7b-8132-1ec6325a1081-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.480898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480827 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63056725-77e3-4c7b-8132-1ec6325a1081-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.480959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480904 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.481017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.480961 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.481090 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.481071 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll754\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-kube-api-access-ll754\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583207 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583229 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583255 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63056725-77e3-4c7b-8132-1ec6325a1081-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583357 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63056725-77e3-4c7b-8132-1ec6325a1081-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583427 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583486 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583627 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583650 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll754\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-kube-api-access-ll754\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583686 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.583808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.583783 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.584193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.584133 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63056725-77e3-4c7b-8132-1ec6325a1081-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.584193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.584146 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.584340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.584190 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.585643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.585625 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.585936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.585919 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63056725-77e3-4c7b-8132-1ec6325a1081-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.593224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.593204 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.593429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.593413 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll754\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-kube-api-access-ll754\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:53.766131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.766095 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:53.791201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:53.791170 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:54.134229 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:49:54.134199 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63056725_77e3_4c7b_8132_1ec6325a1081.slice/crio-50111f0636e874ac64367afbb998b5b80d9365dd54c131167755bfdc593e9692 WatchSource:0}: Error finding container 50111f0636e874ac64367afbb998b5b80d9365dd54c131167755bfdc593e9692: Status 404 returned error can't find the container with id 50111f0636e874ac64367afbb998b5b80d9365dd54c131167755bfdc593e9692 Nov 01 22:49:54.135474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.135452 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99"] Nov 01 22:49:54.137570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.137534 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.137676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.137634 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.137676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.137666 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.347362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.347320 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" event={"ID":"63056725-77e3-4c7b-8132-1ec6325a1081","Type":"ContainerStarted","Data":"68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141"} Nov 01 22:49:54.347568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.347383 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" event={"ID":"63056725-77e3-4c7b-8132-1ec6325a1081","Type":"ContainerStarted","Data":"50111f0636e874ac64367afbb998b5b80d9365dd54c131167755bfdc593e9692"} Nov 01 22:49:54.349353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.349327 4981 generic.go:356] "Generic (PLEG): container finished" podID="2d275241-c066-4ab8-828a-b6acab847630" containerID="b4b94f042fb1eca9188b89ba18bf6bd6cba017a9ba67d22d51dc06c84300f8db" exitCode=0 Nov 01 22:49:54.349501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.349366 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" event={"ID":"2d275241-c066-4ab8-828a-b6acab847630","Type":"ContainerDied","Data":"b4b94f042fb1eca9188b89ba18bf6bd6cba017a9ba67d22d51dc06c84300f8db"} Nov 01 22:49:54.349864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.349847 4981 scope.go:122] "RemoveContainer" containerID="b4b94f042fb1eca9188b89ba18bf6bd6cba017a9ba67d22d51dc06c84300f8db" Nov 01 22:49:54.352200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.352183 4981 generic.go:356] "Generic (PLEG): container finished" podID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerID="cb9dccbfbac2618181f24c84af91df0b6beecd726fe630392cf5847807901a4d" exitCode=0 Nov 01 22:49:54.352299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.352213 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" event={"ID":"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61","Type":"ContainerDied","Data":"cb9dccbfbac2618181f24c84af91df0b6beecd726fe630392cf5847807901a4d"} Nov 01 22:49:54.352456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.352426 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.352536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.352519 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.352617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.352595 4981 scope.go:122] "RemoveContainer" containerID="cb9dccbfbac2618181f24c84af91df0b6beecd726fe630392cf5847807901a4d" Nov 01 22:49:54.352663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.352588 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.354475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.354447 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.354567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.354549 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.354653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.354612 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:49:54.376140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.376015 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podStartSLOduration=1.375974965 podStartE2EDuration="1.375974965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:54.368055799 +0000 UTC m=+1559.795933727" watchObservedRunningTime="2026-11-01 22:49:54.375974965 +0000 UTC m=+1559.803852897" Nov 01 22:49:54.494478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.494427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:54.791502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:54.791465 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:49:55.299121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.299081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:55.551348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.551256 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" event={"ID":"2d275241-c066-4ab8-828a-b6acab847630","Type":"ContainerStarted","Data":"648f23c0a664b56e7839a63dea18b6f9272ba68a35608308128d861a0b57d205"} Nov 01 22:49:55.557159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.557126 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" event={"ID":"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61","Type":"ContainerStarted","Data":"8ec0664be02366b1269f59dc80dcf0d8d1461bb5fe0e8246585a5e1e6c575f04"} Nov 01 22:49:55.704029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.703981 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:49:55.704431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.704318 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 22:49:55.704630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.704604 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 22:49:55.824656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.818338 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:49:55.824656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.818422 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 22:49:55.900042 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:55.900009 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:49:56.529633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.527792 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:49:56.529633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.527934 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:49:56.529633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.527978 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 22:49:56.731629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.719895 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 22:49:56.731629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.719955 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 22:49:56.793402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.792869 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:49:56.793402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.792926 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 22:49:56.800665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:56.800598 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:57.524600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:57.523671 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:49:57.524600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:57.523756 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 22:49:57.670452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:57.670418 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:57.702600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:57.702140 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 22:49:57.702600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:57.702204 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 22:49:57.803093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:57.802996 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:49:57.804737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:57.804699 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 22:49:58.266593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.265816 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:58.532687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.529306 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:49:58.532687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.529381 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 22:49:58.583221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.583183 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:58.736609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.729097 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:58.737052 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.737021 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 22:49:58.745600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.737239 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 22:49:58.797268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.795007 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:49:58.797268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:58.795092 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 22:49:59.273603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.223559 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:59.523024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.522986 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:49:59.523313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.523285 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 22:49:59.595244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.595154 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:49:59.705061 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.704987 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 22:49:59.705372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.705337 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 22:49:59.798249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.798214 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:49:59.798538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:49:59.798512 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 22:50:00.015607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.013278 4981 scope.go:122] "RemoveContainer" containerID="a18c6eb194dda33227d051f65ad778e86859734fa26afcf4ff5a8abc2677c577" Nov 01 22:50:00.110159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.110114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:00.536455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.534040 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:50:00.536455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.534102 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 22:50:00.712955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.712921 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 22:50:00.713371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.713329 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 22:50:00.803766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.803724 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:50:00.804245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.804211 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 22:50:00.858197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:00.858129 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:01.557911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:01.557867 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 01 22:50:01.558478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:01.557937 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 01 22:50:01.665988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:01.665951 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:01.700597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:01.699983 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 01 22:50:01.700597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:01.700110 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 01 22:50:01.798612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:01.797023 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 01 22:50:01.798612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:01.797104 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 01 22:50:02.042356 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.042317 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q"] Nov 01 22:50:02.051366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.051297 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980bfa4e-txk8q"] Nov 01 22:50:02.446723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.446682 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:02.527602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.527517 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:50:02.532598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.530632 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:50:02.705512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.705410 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:50:02.707284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.707261 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:50:02.708904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.708879 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:50:02.741734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.741692 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2754c0df-eb00-4688-b7eb-112aa189c0fa" path="/var/lib/kubelet/pods/2754c0df-eb00-4688-b7eb-112aa189c0fa/volumes" Nov 01 22:50:02.799100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.799060 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:50:02.800882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.800854 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:50:02.801561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:02.801533 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:50:03.270009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:03.269978 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:03.957372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:03.957341 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:04.700238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:04.700209 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:05.363434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:05.363393 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:06.148342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:06.148252 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:07.106342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:07.106311 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:08.271164 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:08.271132 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:09.071763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:09.071726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:09.620535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:09.620477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:09.724822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:09.724791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:09.838601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:09.836500 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:10.047522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.047490 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:10.160827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.160789 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:10.178033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.178000 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86"] Nov 01 22:50:10.203790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.203754 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e36490-2mz86"] Nov 01 22:50:10.257866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.257814 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:10.611091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.611014 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:10.811798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.811757 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd314c4-7cba-4ac9-9bc7-5e80de063f8b" path="/var/lib/kubelet/pods/8bd314c4-7cba-4ac9-9bc7-5e80de063f8b/volumes" Nov 01 22:50:10.846520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:10.846482 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:11.096668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:11.096633 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:11.580922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:11.580857 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:11.885487 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:11.885376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:12.742725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:12.742681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:13.476365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:13.476309 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:14.147536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:14.147488 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:14.250924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:14.250691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:14.528502 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:50:14.528458 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:50:14.955055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:14.955007 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:15.654451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:15.654417 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:16.381807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:16.381771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:17.089973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.089937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:17.218440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.218409 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:17.424526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.424490 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9"] Nov 01 22:50:17.424852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.424823 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" containerID="cri-o://271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe" gracePeriod=30 Nov 01 22:50:17.450410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.450375 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv"] Nov 01 22:50:17.450674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.450632 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" containerID="cri-o://24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8" gracePeriod=30 Nov 01 22:50:17.455652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.455555 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:17.456887 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.456861 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft"] Nov 01 22:50:17.457102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:17.457080 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" containerID="cri-o://a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31" gracePeriod=30 Nov 01 22:50:18.160419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:18.160388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:18.366306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:18.366263 4981 patch_prober.go:29] interesting pod/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 01 22:50:18.366492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:18.366333 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:50:18.397865 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:18.397819 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:18.735858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:18.735823 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:19.136656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:19.136406 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:19.456185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:19.456152 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:20.371732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:20.371690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:21.149424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:21.149389 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:21.921069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:21.921032 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:22.618712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.617070 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:22.848772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.848745 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:50:22.893454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.893418 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv"] Nov 01 22:50:22.895005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.894981 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" Nov 01 22:50:22.895005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.895005 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" Nov 01 22:50:22.895244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.895232 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerName="istio-proxy" Nov 01 22:50:22.902319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.901673 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:22.910249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.910225 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-39e952-openshift-default-dockercfg-xclwz\"" Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.911986 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-podinfo\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912102 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-token\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-token\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912145 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-certs\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912177 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-socket\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912252 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19d413e-0380-4383-968c-d8de43fc8dd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19d413e-0380-4383-968c-d8de43fc8dd9-istiod-ca-cert\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912401 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-kube-api-access-f45pf\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-kube-api-access-f45pf\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912456 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-credential-socket\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912548 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-data\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.913455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.912737 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-envoy\") pod \"b19d413e-0380-4383-968c-d8de43fc8dd9\" (UID: \"b19d413e-0380-4383-968c-d8de43fc8dd9\") " Nov 01 22:50:22.915308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.915147 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19d413e-0380-4383-968c-d8de43fc8dd9-istiod-ca-cert" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:22.915308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.915258 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-token" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:22.915481 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.915405 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-certs" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.915771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.915590 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-socket" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.916081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.915900 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-credential-socket" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.916234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.916204 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-data" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.920877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.920853 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv"] Nov 01 22:50:22.924390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.921343 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-kube-api-access-f45pf" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "kube-api-access-f45pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:22.926527 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.926502 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-envoy" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:22.934250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.934204 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-podinfo" pod "b19d413e-0380-4383-968c-d8de43fc8dd9" (UID: "b19d413e-0380-4383-968c-d8de43fc8dd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:22.972798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.972769 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:50:22.977666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:22.977636 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:50:23.015174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015145 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istiod-ca-cert\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015214 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9efbf218-9374-41a1-bcc3-00b296a7b05d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efbf218-9374-41a1-bcc3-00b296a7b05d-istiod-ca-cert\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015254 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-credential-socket\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015288 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-certs\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015383 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-token\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-token\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015411 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-certs\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015475 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-envoy\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015514 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-data\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015541 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-socket\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015566 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-podinfo\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015678 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-kube-api-access-g778f\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-kube-api-access-g778f\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015722 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-kube-api-access-5n5w5\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-kube-api-access-5n5w5\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.015803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015796 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-credential-socket\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015881 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-token\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-token\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015912 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-podinfo\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015930 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-socket\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.015960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.015952 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-data\") pod \"9efbf218-9374-41a1-bcc3-00b296a7b05d\" (UID: \"9efbf218-9374-41a1-bcc3-00b296a7b05d\") " Nov 01 22:50:23.016065 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016004 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-envoy\") pod \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\" (UID: \"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa\") " Nov 01 22:50:23.016340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016322 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-istio-token\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016474 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-credential-socket\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016556 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86k5t\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-kube-api-access-86k5t\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016651 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-socket\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016680 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a20f96-6209-41dd-add8-de189e2fba0c-istiod-ca-cert\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016735 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-envoy\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016754 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-data\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016829 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-certs\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.016903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016876 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a20f96-6209-41dd-add8-de189e2fba0c-istio-podinfo\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.017022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016981 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.016992 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017001 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017015 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017029 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19d413e-0380-4383-968c-d8de43fc8dd9-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017041 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f45pf\" (UniqueName: \"kubernetes.io/projected/b19d413e-0380-4383-968c-d8de43fc8dd9-kube-api-access-f45pf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017052 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017062 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017070 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19d413e-0380-4383-968c-d8de43fc8dd9-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.017407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017387 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istiod-ca-cert" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:23.017688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017674 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efbf218-9374-41a1-bcc3-00b296a7b05d-istiod-ca-cert" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:50:23.017892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.017878 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-credential-socket" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.018023 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.018011 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-certs" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.018197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.018174 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t"] Nov 01 22:50:23.018496 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.018470 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-socket" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.018664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.018597 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-credential-socket" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.018845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.018816 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-data" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.019550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019519 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" Nov 01 22:50:23.019550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019540 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" Nov 01 22:50:23.019739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019608 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" Nov 01 22:50:23.019739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019616 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" Nov 01 22:50:23.019739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019643 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" Nov 01 22:50:23.019739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019651 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" Nov 01 22:50:23.019973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019888 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" Nov 01 22:50:23.019973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019909 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerName="istio-proxy" Nov 01 22:50:23.019973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.019923 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerName="istio-proxy" Nov 01 22:50:23.022848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.020540 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-token" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:23.022848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.021504 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-kube-api-access-g778f" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "kube-api-access-g778f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:23.022848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.021812 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-data" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.022848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.021985 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-socket" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.022848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.022143 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-certs" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.025281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.024127 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-envoy" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.025281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.025103 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-envoy" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:50:23.031609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.025469 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-podinfo" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:23.031609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.026978 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.038849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.038807 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-kube-api-access-5n5w5" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "kube-api-access-5n5w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:23.046368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.046338 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t"] Nov 01 22:50:23.048803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.048777 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-podinfo" pod "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" (UID: "3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:50:23.049623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.049597 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-token" pod "9efbf218-9374-41a1-bcc3-00b296a7b05d" (UID: "9efbf218-9374-41a1-bcc3-00b296a7b05d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:50:23.110643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.110537 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b48ab8-openshift-default-dockercfg-2r4pn\"" Nov 01 22:50:23.119034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.118999 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k5t\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-kube-api-access-86k5t\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.119256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119056 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-socket\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.119256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119229 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a20f96-6209-41dd-add8-de189e2fba0c-istiod-ca-cert\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.119386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119295 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-data\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.119386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119328 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-credential-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.119386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119375 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-envoy\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.119564 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119499 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-data\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.119679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119655 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-certs\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.119745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119723 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a20f96-6209-41dd-add8-de189e2fba0c-istio-podinfo\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.119803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119757 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d827f366-8ae1-425e-98ab-40ed8943f512-istio-podinfo\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.119840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119817 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-istio-token\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.119876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.119848 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-data\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.120037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120013 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-certs\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.120114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120032 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a20f96-6209-41dd-add8-de189e2fba0c-istiod-ca-cert\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.120175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120108 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-envoy\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.120175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120144 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d827f366-8ae1-425e-98ab-40ed8943f512-istiod-ca-cert\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.120284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120177 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-istio-token\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.120284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120256 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-socket\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.120394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120376 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndj6x\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-kube-api-access-ndj6x\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.120472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120452 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.120532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120487 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-credential-socket\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.120692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120632 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-certs\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.120850 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120832 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9efbf218-9374-41a1-bcc3-00b296a7b05d-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.120913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120847 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-credential-socket\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.120913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120856 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.120913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120872 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.120913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120887 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.120913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120901 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120919 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120933 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120947 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120962 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120978 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g778f\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-kube-api-access-g778f\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.120993 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n5w5\" (UniqueName: \"kubernetes.io/projected/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-kube-api-access-5n5w5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.121008 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.121022 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.121048 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.121065 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.121081 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9efbf218-9374-41a1-bcc3-00b296a7b05d-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.121097 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.121179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.121110 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:50:23.122378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.122355 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-envoy\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.122875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.122837 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a20f96-6209-41dd-add8-de189e2fba0c-istio-podinfo\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.129396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.129364 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86k5t\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-kube-api-access-86k5t\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.129507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.129492 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-istio-token\") pod \"e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.222654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.222612 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d827f366-8ae1-425e-98ab-40ed8943f512-istio-podinfo\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.222833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.222684 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-istio-token\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.222901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.222848 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-envoy\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.222901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.222883 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d827f366-8ae1-425e-98ab-40ed8943f512-istiod-ca-cert\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223038 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223013 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndj6x\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-kube-api-access-ndj6x\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223077 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223129 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-certs\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223268 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-data\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223348 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-credential-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223515 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223552 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-certs\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223755 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-data\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223777 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d827f366-8ae1-425e-98ab-40ed8943f512-istiod-ca-cert\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.223856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.223801 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-credential-socket\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.225030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.225007 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-envoy\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.225135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.225117 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d827f366-8ae1-425e-98ab-40ed8943f512-istio-podinfo\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.230986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.230958 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-istio-token\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.231168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.231149 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndj6x\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-kube-api-access-ndj6x\") pod \"e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.267715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.267683 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:23.311934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.311898 4981 generic.go:356] "Generic (PLEG): container finished" podID="b19d413e-0380-4383-968c-d8de43fc8dd9" containerID="271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe" exitCode=0 Nov 01 22:50:23.312095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.311951 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" Nov 01 22:50:23.312095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.311976 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" event={"ID":"b19d413e-0380-4383-968c-d8de43fc8dd9","Type":"ContainerDied","Data":"271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe"} Nov 01 22:50:23.312095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.312041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9" event={"ID":"b19d413e-0380-4383-968c-d8de43fc8dd9","Type":"ContainerDied","Data":"f10c4bbee8d869c42ba7a0998b26cecf8c1061841d0b418a79f4c54d08617eb4"} Nov 01 22:50:23.312095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.312068 4981 scope.go:122] "RemoveContainer" containerID="271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe" Nov 01 22:50:23.314914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.314884 4981 generic.go:356] "Generic (PLEG): container finished" podID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" containerID="24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8" exitCode=0 Nov 01 22:50:23.315044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.314932 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" event={"ID":"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa","Type":"ContainerDied","Data":"24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8"} Nov 01 22:50:23.315044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.315019 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" event={"ID":"3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa","Type":"ContainerDied","Data":"66609db549b344480a868becacddd693ad62bbf65ba0454655ca13f2dac61175"} Nov 01 22:50:23.315190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.314946 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv" Nov 01 22:50:23.317829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.317804 4981 generic.go:356] "Generic (PLEG): container finished" podID="9efbf218-9374-41a1-bcc3-00b296a7b05d" containerID="a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31" exitCode=0 Nov 01 22:50:23.317952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.317853 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" event={"ID":"9efbf218-9374-41a1-bcc3-00b296a7b05d","Type":"ContainerDied","Data":"a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31"} Nov 01 22:50:23.317952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.317893 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" event={"ID":"9efbf218-9374-41a1-bcc3-00b296a7b05d","Type":"ContainerDied","Data":"cc761db51f8b1caa903cdde6eee8b9fbbcc4dd8a94214991aa6ee68ad95131a9"} Nov 01 22:50:23.317952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.317920 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft" Nov 01 22:50:23.327036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.327014 4981 scope.go:122] "RemoveContainer" containerID="271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe" Nov 01 22:50:23.327536 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:50:23.327508 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe\": container with ID starting with 271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe not found: ID does not exist" containerID="271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe" Nov 01 22:50:23.327674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.327546 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe"} err="failed to get container status \"271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe\": rpc error: code = NotFound desc = could not find container \"271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe\": container with ID starting with 271e01d856564c804774469909cb9393b80a9d5f5a5d1890b95c61fce38addfe not found: ID does not exist" Nov 01 22:50:23.327674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.327625 4981 scope.go:122] "RemoveContainer" containerID="24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8" Nov 01 22:50:23.348367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.348334 4981 scope.go:122] "RemoveContainer" containerID="24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8" Nov 01 22:50:23.349439 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:50:23.349412 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8\": container with ID starting with 24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8 not found: ID does not exist" containerID="24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8" Nov 01 22:50:23.349553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.349451 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8"} err="failed to get container status \"24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8\": rpc error: code = NotFound desc = could not find container \"24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8\": container with ID starting with 24e77f363fa8040262587de0ed0c68d1402233d1f311475616e0ab5286bb1dd8 not found: ID does not exist" Nov 01 22:50:23.349553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.349484 4981 scope.go:122] "RemoveContainer" containerID="a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31" Nov 01 22:50:23.357184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.357097 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9"] Nov 01 22:50:23.364336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.364284 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-p4mq9"] Nov 01 22:50:23.372317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.372293 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv"] Nov 01 22:50:23.373674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.372943 4981 scope.go:122] "RemoveContainer" containerID="3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7" Nov 01 22:50:23.374721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.374680 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:23.384554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.384527 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-7bngv"] Nov 01 22:50:23.402022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.401992 4981 scope.go:122] "RemoveContainer" containerID="a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31" Nov 01 22:50:23.402379 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:50:23.402355 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31\": container with ID starting with a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31 not found: ID does not exist" containerID="a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31" Nov 01 22:50:23.402491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.402386 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31"} err="failed to get container status \"a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31\": rpc error: code = NotFound desc = could not find container \"a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31\": container with ID starting with a4fd482361750520dcd5996014c30593871ede435a4e2b752b9e6da1c865ff31 not found: ID does not exist" Nov 01 22:50:23.402491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.402404 4981 scope.go:122] "RemoveContainer" containerID="3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7" Nov 01 22:50:23.403192 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:50:23.403105 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7\": container with ID starting with 3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7 not found: ID does not exist" containerID="3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7" Nov 01 22:50:23.403192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.403160 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7"} err="failed to get container status \"3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7\": rpc error: code = NotFound desc = could not find container \"3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7\": container with ID starting with 3d6eb4ec351d019ea198b926cb409043aa1ad533ae9a9af1c7b1db017e4413c7 not found: ID does not exist" Nov 01 22:50:23.407628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.407528 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft"] Nov 01 22:50:23.407973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.407940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:23.413165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.413145 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e36490-openshift-default-5bbb8d86d9-jnlft"] Nov 01 22:50:23.588532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.587777 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv"] Nov 01 22:50:23.591888 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:50:23.591855 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a20f96_6209_41dd_add8_de189e2fba0c.slice/crio-b83dc9a14c9cc070bc61d7239bd16f81c0a5529da49f3fca85519fba4fd96c1f WatchSource:0}: Error finding container b83dc9a14c9cc070bc61d7239bd16f81c0a5529da49f3fca85519fba4fd96c1f: Status 404 returned error can't find the container with id b83dc9a14c9cc070bc61d7239bd16f81c0a5529da49f3fca85519fba4fd96c1f Nov 01 22:50:23.597984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.597937 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:50:23.598109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.598041 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:50:23.598109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.598088 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:50:23.733605 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:50:23.733559 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd827f366_8ae1_425e_98ab_40ed8943f512.slice/crio-5a0f948dd239a0c6bafa9faf1772eb39b1917c7bb90119bc9e7ef69fcc36d725 WatchSource:0}: Error finding container 5a0f948dd239a0c6bafa9faf1772eb39b1917c7bb90119bc9e7ef69fcc36d725: Status 404 returned error can't find the container with id 5a0f948dd239a0c6bafa9faf1772eb39b1917c7bb90119bc9e7ef69fcc36d725 Nov 01 22:50:23.735568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.735545 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t"] Nov 01 22:50:23.737679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.737650 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:50:23.737757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.737714 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:50:23.737757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:23.737752 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:50:24.161334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.161301 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:24.334645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.332933 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" event={"ID":"42a20f96-6209-41dd-add8-de189e2fba0c","Type":"ContainerStarted","Data":"f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0"} Nov 01 22:50:24.334645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.333048 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" event={"ID":"42a20f96-6209-41dd-add8-de189e2fba0c","Type":"ContainerStarted","Data":"b83dc9a14c9cc070bc61d7239bd16f81c0a5529da49f3fca85519fba4fd96c1f"} Nov 01 22:50:24.338491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.338443 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" event={"ID":"d827f366-8ae1-425e-98ab-40ed8943f512","Type":"ContainerStarted","Data":"ceeeba62efcebe1e993adc8be6279e3560a6247ebadf68e4071f33189dd642bf"} Nov 01 22:50:24.338741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.338505 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" event={"ID":"d827f366-8ae1-425e-98ab-40ed8943f512","Type":"ContainerStarted","Data":"5a0f948dd239a0c6bafa9faf1772eb39b1917c7bb90119bc9e7ef69fcc36d725"} Nov 01 22:50:24.362411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.362359 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podStartSLOduration=25.362343233 podStartE2EDuration="25.362343233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:24.355691124 +0000 UTC m=+1589.783569070" watchObservedRunningTime="2026-11-01 22:50:24.362343233 +0000 UTC m=+1589.790221164" Nov 01 22:50:24.376018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.375985 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:24.382713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.382680 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:24.385010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.384966 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podStartSLOduration=14.384951471 podStartE2EDuration="14.384951471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:24.379857287 +0000 UTC m=+1589.807735255" watchObservedRunningTime="2026-11-01 22:50:24.384951471 +0000 UTC m=+1589.812829401" Nov 01 22:50:24.738498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.738468 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa" path="/var/lib/kubelet/pods/3a6f6927-7cd2-44d1-bc7f-b2bf471a83fa/volumes" Nov 01 22:50:24.739082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.739066 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9efbf218-9374-41a1-bcc3-00b296a7b05d" path="/var/lib/kubelet/pods/9efbf218-9374-41a1-bcc3-00b296a7b05d/volumes" Nov 01 22:50:24.739548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.739534 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19d413e-0380-4383-968c-d8de43fc8dd9" path="/var/lib/kubelet/pods/b19d413e-0380-4383-968c-d8de43fc8dd9/volumes" Nov 01 22:50:24.974163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:24.974133 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:25.267888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:25.267855 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:25.274534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:25.274510 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:25.354338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:25.354303 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:25.354543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:25.354458 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:25.355295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:25.355275 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:50:25.355410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:25.355394 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:50:26.139608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:26.136136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:26.464388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:26.464358 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:26.571909 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:26.571872 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:26.712977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:26.712941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:26.938997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:26.938954 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:27.054131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:27.054099 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:27.243879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:27.232412 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:28.097917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.097880 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:28.102830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.102801 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9"] Nov 01 22:50:28.110777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.110742 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.119502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.119472 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c49240\"" Nov 01 22:50:28.123879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.123502 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9"] Nov 01 22:50:28.130477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.130446 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk"] Nov 01 22:50:28.136115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.136093 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.156382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.153893 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk"] Nov 01 22:50:28.162972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.162944 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c49240-dockercfg-wfn8s\"" Nov 01 22:50:28.199400 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.199354 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqd2\" (UniqueName: \"kubernetes.io/projected/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-kube-api-access-lgqd2\") pod \"payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.199630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.199536 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c38bb28-8f0c-4352-bc91-a259731f3c9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.199814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.199786 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.199893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.199865 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzns\" (UniqueName: \"kubernetes.io/projected/5c38bb28-8f0c-4352-bc91-a259731f3c9a-kube-api-access-ptzns\") pod \"payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.223336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.219920 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:28.312600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.309985 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqd2\" (UniqueName: \"kubernetes.io/projected/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-kube-api-access-lgqd2\") pod \"payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.312600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.310102 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c38bb28-8f0c-4352-bc91-a259731f3c9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.312600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.310255 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.312600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.310317 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzns\" (UniqueName: \"kubernetes.io/projected/5c38bb28-8f0c-4352-bc91-a259731f3c9a-kube-api-access-ptzns\") pod \"payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.312600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.311948 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c38bb28-8f0c-4352-bc91-a259731f3c9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.313338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.312736 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.404646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.403285 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqd2\" (UniqueName: \"kubernetes.io/projected/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-kube-api-access-lgqd2\") pod \"payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.423598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.423313 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzns\" (UniqueName: \"kubernetes.io/projected/5c38bb28-8f0c-4352-bc91-a259731f3c9a-kube-api-access-ptzns\") pod \"payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.456598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.455136 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:28.480600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.476203 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:28.732621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.731664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:28.914495 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:50:28.914468 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e75445e_7d5c_4ee1_86b7_9c0a6b874572.slice/crio-381c89d8c64dff44863fe9f9771f00da22b816a07dd969b0f9e4eedc98ba9894 WatchSource:0}: Error finding container 381c89d8c64dff44863fe9f9771f00da22b816a07dd969b0f9e4eedc98ba9894: Status 404 returned error can't find the container with id 381c89d8c64dff44863fe9f9771f00da22b816a07dd969b0f9e4eedc98ba9894 Nov 01 22:50:28.916939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.916914 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9"] Nov 01 22:50:28.937987 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:50:28.937917 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c38bb28_8f0c_4352_bc91_a259731f3c9a.slice/crio-dfa8f0e23e0e142d888a9cfbdb70da6d99b026c77a091fe0f16fac11f7396c35 WatchSource:0}: Error finding container dfa8f0e23e0e142d888a9cfbdb70da6d99b026c77a091fe0f16fac11f7396c35: Status 404 returned error can't find the container with id dfa8f0e23e0e142d888a9cfbdb70da6d99b026c77a091fe0f16fac11f7396c35 Nov 01 22:50:28.940233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:28.940205 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk"] Nov 01 22:50:29.161399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.161362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:29.219474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.219383 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv"] Nov 01 22:50:29.224753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.224727 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.232827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.232799 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b48ab8\"" Nov 01 22:50:29.233768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.233745 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv"] Nov 01 22:50:29.284381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.284340 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5"] Nov 01 22:50:29.294407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.294379 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:29.298965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.298901 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5"] Nov 01 22:50:29.299340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.299320 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b48ab8-dockercfg-hv7hc\"" Nov 01 22:50:29.323599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.317885 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6mm\" (UniqueName: \"kubernetes.io/projected/def232dc-e82c-4e0c-851a-bca840f18505-kube-api-access-xh6mm\") pod \"payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.323599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.318054 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def232dc-e82c-4e0c-851a-bca840f18505-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.436629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.431140 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwkx\" (UniqueName: \"kubernetes.io/projected/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-kube-api-access-zrwkx\") pod \"payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:29.657010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.656912 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:29.688613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.657526 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:29.691019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.690987 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6mm\" (UniqueName: \"kubernetes.io/projected/def232dc-e82c-4e0c-851a-bca840f18505-kube-api-access-xh6mm\") pod \"payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.692735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.692387 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def232dc-e82c-4e0c-851a-bca840f18505-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.694425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.693729 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def232dc-e82c-4e0c-851a-bca840f18505-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.734929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.734569 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" event={"ID":"5c38bb28-8f0c-4352-bc91-a259731f3c9a","Type":"ContainerStarted","Data":"96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea"} Nov 01 22:50:29.734929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.734676 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" event={"ID":"5c38bb28-8f0c-4352-bc91-a259731f3c9a","Type":"ContainerStarted","Data":"dfa8f0e23e0e142d888a9cfbdb70da6d99b026c77a091fe0f16fac11f7396c35"} Nov 01 22:50:29.734929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.734761 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:29.737361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.737328 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" event={"ID":"2e75445e-7d5c-4ee1-86b7-9c0a6b874572","Type":"ContainerStarted","Data":"09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f"} Nov 01 22:50:29.737568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.737548 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" event={"ID":"2e75445e-7d5c-4ee1-86b7-9c0a6b874572","Type":"ContainerStarted","Data":"381c89d8c64dff44863fe9f9771f00da22b816a07dd969b0f9e4eedc98ba9894"} Nov 01 22:50:29.737829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.737624 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6mm\" (UniqueName: \"kubernetes.io/projected/def232dc-e82c-4e0c-851a-bca840f18505-kube-api-access-xh6mm\") pod \"payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.737947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.737930 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:29.761028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.760968 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" podStartSLOduration=1.760951047 podStartE2EDuration="1.760951047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:29.755103951 +0000 UTC m=+1595.182981883" watchObservedRunningTime="2026-11-01 22:50:29.760951047 +0000 UTC m=+1595.188828978" Nov 01 22:50:29.779688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.779633 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" podStartSLOduration=1.7796147979999999 podStartE2EDuration="1.779614798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:29.773286133 +0000 UTC m=+1595.201164063" watchObservedRunningTime="2026-11-01 22:50:29.779614798 +0000 UTC m=+1595.207492730" Nov 01 22:50:29.795935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.795902 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwkx\" (UniqueName: \"kubernetes.io/projected/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-kube-api-access-zrwkx\") pod \"payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:29.796109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.796053 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:29.796885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.796857 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:29.804836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.804816 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwkx\" (UniqueName: \"kubernetes.io/projected/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-kube-api-access-zrwkx\") pod \"payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:29.893875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.893841 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:29.907774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.907689 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:29.991781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:29.991741 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:30.035647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.035613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:30.038745 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.038715 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g"] Nov 01 22:50:30.047195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.047162 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.051936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.051907 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-39e952-dockercfg-5rwb8\"" Nov 01 22:50:30.052285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.052268 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39e952\"" Nov 01 22:50:30.056678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.055003 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g"] Nov 01 22:50:30.206622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.201290 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb84k\" (UniqueName: \"kubernetes.io/projected/fdfcc93a-a688-497a-aabf-801bd24d2159-kube-api-access-fb84k\") pod \"payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.206622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.201413 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfcc93a-a688-497a-aabf-801bd24d2159-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.495129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.333191 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb84k\" (UniqueName: \"kubernetes.io/projected/fdfcc93a-a688-497a-aabf-801bd24d2159-kube-api-access-fb84k\") pod \"payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.495129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.436929 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfcc93a-a688-497a-aabf-801bd24d2159-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.495129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.438463 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfcc93a-a688-497a-aabf-801bd24d2159-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.526618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.526551 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv"] Nov 01 22:50:30.539844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.539806 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb84k\" (UniqueName: \"kubernetes.io/projected/fdfcc93a-a688-497a-aabf-801bd24d2159-kube-api-access-fb84k\") pod \"payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.633857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.633822 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:30.751547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.751454 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" event={"ID":"def232dc-e82c-4e0c-851a-bca840f18505","Type":"ContainerStarted","Data":"5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1"} Nov 01 22:50:30.751547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.751516 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" event={"ID":"def232dc-e82c-4e0c-851a-bca840f18505","Type":"ContainerStarted","Data":"16a3a16b8e1961f4ce1bc3b3869d6dbdbf5d8ee05b4204f3e6690f279320b572"} Nov 01 22:50:30.751547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.751546 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:30.777188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.777120 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" podStartSLOduration=1.7770683539999999 podStartE2EDuration="1.777068354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:30.77360325 +0000 UTC m=+1596.201481181" watchObservedRunningTime="2026-11-01 22:50:30.777068354 +0000 UTC m=+1596.204946287" Nov 01 22:50:30.786498 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:50:30.786468 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffba04c8_57c1_4023_9e7d_5b7cc8e6c60e.slice/crio-b3faf24466bbfa1139ca86bdba8dc37de41baa6a5185aa0e21a81747b486221a WatchSource:0}: Error finding container b3faf24466bbfa1139ca86bdba8dc37de41baa6a5185aa0e21a81747b486221a: Status 404 returned error can't find the container with id b3faf24466bbfa1139ca86bdba8dc37de41baa6a5185aa0e21a81747b486221a Nov 01 22:50:30.788644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.788615 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5"] Nov 01 22:50:30.988882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:30.988853 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g"] Nov 01 22:50:30.989378 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:50:30.989357 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfcc93a_a688_497a_aabf_801bd24d2159.slice/crio-129e986f8697ba581eefcfd837a0e7726cad88d12b83a33d4cfa263e9b8aa791 WatchSource:0}: Error finding container 129e986f8697ba581eefcfd837a0e7726cad88d12b83a33d4cfa263e9b8aa791: Status 404 returned error can't find the container with id 129e986f8697ba581eefcfd837a0e7726cad88d12b83a33d4cfa263e9b8aa791 Nov 01 22:50:31.015314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.015234 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:31.763952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.763907 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" event={"ID":"fdfcc93a-a688-497a-aabf-801bd24d2159","Type":"ContainerStarted","Data":"bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264"} Nov 01 22:50:31.764414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.763988 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:31.764414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.764017 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" event={"ID":"fdfcc93a-a688-497a-aabf-801bd24d2159","Type":"ContainerStarted","Data":"129e986f8697ba581eefcfd837a0e7726cad88d12b83a33d4cfa263e9b8aa791"} Nov 01 22:50:31.765753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.765723 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" event={"ID":"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e","Type":"ContainerStarted","Data":"42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a"} Nov 01 22:50:31.765882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.765776 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" event={"ID":"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e","Type":"ContainerStarted","Data":"b3faf24466bbfa1139ca86bdba8dc37de41baa6a5185aa0e21a81747b486221a"} Nov 01 22:50:31.765882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.765809 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:31.783760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.783708 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" podStartSLOduration=1.78369388 podStartE2EDuration="1.78369388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:31.781599992 +0000 UTC m=+1597.209477920" watchObservedRunningTime="2026-11-01 22:50:31.78369388 +0000 UTC m=+1597.211571810" Nov 01 22:50:31.793243 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.793206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:31.804328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:31.804279 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" podStartSLOduration=2.804264072 podStartE2EDuration="2.804264072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:31.800834079 +0000 UTC m=+1597.228712011" watchObservedRunningTime="2026-11-01 22:50:31.804264072 +0000 UTC m=+1597.232142003" Nov 01 22:50:32.554805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:32.554775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:33.287385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:33.287354 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:34.061313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:34.061277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:34.684861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:34.684827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:35.698827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:35.698795 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:36.071193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:36.071099 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz"] Nov 01 22:50:36.076840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:36.076807 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-534f6d86-lfmrz"] Nov 01 22:50:36.737401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:36.737359 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f" path="/var/lib/kubelet/pods/3df6aa13-4de6-4fc3-b3dd-fc96a09bb77f/volumes" Nov 01 22:50:36.996084 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:36.995993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:38.102616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:38.102537 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:39.111105 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:39.111068 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:39.212473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:39.212439 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:40.510005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:40.509968 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:40.754861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:40.754831 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 22:50:40.755009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:40.754891 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 22:50:41.198945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:41.198913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:41.705046 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:41.705008 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:41.770131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:41.770100 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 22:50:42.562846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:42.562811 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:42.778735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:42.778704 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 22:50:42.779153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:42.778786 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 22:50:43.378396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:43.378364 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:44.170209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:44.170164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:44.531434 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:50:44.531395 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:50:44.967793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:44.967762 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:45.756193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:45.756163 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:46.642783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:46.642750 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:47.356722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:47.356691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:48.165495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:48.165452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:48.796968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:48.796917 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:49.536518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:49.536478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:50.269213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:50.269184 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:50.366238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:50.365855 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:50.639459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:50.639375 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:51.386028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:51.385989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:51.501840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:51.501802 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:51.735293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:51.735264 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:52.490832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:52.490796 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:52.612004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:52.611043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:52.784968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:52.784879 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:53.467399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:53.467365 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:54.246199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:54.246111 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:54.970532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:54.970488 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:55.764168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:55.764132 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:56.482944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:56.482912 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:57.164735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:57.164698 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:57.618000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:57.617967 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:58.167959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:58.167925 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:58.953659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:58.953604 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:50:59.926617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:50:59.926541 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:00.492315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:00.492285 4981 scope.go:122] "RemoveContainer" containerID="59e330a30be34e929f9da551f277acdccb1ae111e40bc3f1ac981c855b97ae52" Nov 01 22:51:00.504120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:00.504096 4981 scope.go:122] "RemoveContainer" containerID="8911d152f0481c48cb6c6e80a6ad5aff5d9f20cd74f9cc68d9c9d6d9808909ad" Nov 01 22:51:00.515471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:00.515446 4981 scope.go:122] "RemoveContainer" containerID="b0e48db0cf269d874bb94aca2cfc2f64e95a4bd33271e3986ba379e634c44575" Nov 01 22:51:00.993340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:00.993311 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:01.765224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:01.765187 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:02.110514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:02.110421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:02.215552 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:02.215520 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:02.699262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:02.699226 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:03.480011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:03.479975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:04.270927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:04.270893 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:05.262176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:05.262096 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:05.928474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:05.928444 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:06.678852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:06.678819 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:07.540902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:07.540861 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:08.326618 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:08.326569 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:08.434856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:08.434814 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:08.537827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:08.537796 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:08.972497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:08.972455 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:09.829722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:09.829691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:10.580514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:10.580478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:11.206363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:11.206332 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:12.039260 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:12.039232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:12.741971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:12.741939 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:13.411793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:13.411757 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:14.076206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:14.076171 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:14.244477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:14.244443 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:14.527414 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:51:14.527379 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:51:14.833394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:14.833249 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:15.136967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:15.136884 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:15.243239 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:15.243199 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:15.562605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:15.562562 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:16.363029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:16.362996 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:17.066811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:17.066754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:17.675022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:17.674990 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:18.446425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:18.446394 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:19.235118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:19.235078 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:19.777961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:19.777924 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:20.526883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:20.526851 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:21.242878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:21.242843 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:22.027407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:22.027370 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:22.655352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:22.655322 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:23.289788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:23.289756 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:24.143451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:24.143419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:24.902459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:24.902422 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:25.585292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:25.585251 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:26.300280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:26.300245 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:26.977009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:26.976978 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:27.833404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:27.833374 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:28.623963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:28.623881 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:29.417569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:29.417537 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:30.176830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:30.176798 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:30.781997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:30.781961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:31.465689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:31.465660 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:32.155917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:32.155879 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:32.817461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:32.817415 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:33.908336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:33.908305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:35.759530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:35.759495 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:36.027662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.020477 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:51:36.041100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.040163 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.050609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.049657 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.067655 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.067613 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:51:36.068210 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.068183 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:51:36.069621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.069593 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.094694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.093712 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.127746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.126747 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.127746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.127308 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:51:36.223332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.223288 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.578610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.549565 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded" Nov 01 22:51:36.631501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.631449 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.806673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.783704 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:36.874761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.874397 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 01 22:51:36.897819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.897757 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:51:36.935784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:36.935735 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:37.084665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.084607 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:51:37.133508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.124347 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:51:37.158922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.154956 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:51:37.191147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.191106 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:37.194919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.194882 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:51:37.195220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.195204 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:51:37.195345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.195335 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:51:37.196266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.196223 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:37.197211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.197180 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:51:37.228382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.227709 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:51:37.230548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.228875 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:51:37.551550 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.551494 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:51:37.551777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.551666 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:51:37.551777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.551722 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:51:37.551777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.551771 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:51:37.551950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.551881 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:51:37.992247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:37.992186 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:38.822364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:38.822303 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:39.682603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:39.679149 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:40.407989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:40.407920 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:41.223529 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:41.223495 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:41.949281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:41.949248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:42.762001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:42.761949 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:43.379077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:43.379041 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:44.246625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:44.242114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:44.537684 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:51:44.537545 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:51:44.818346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:44.818254 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:45.446959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:45.446922 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:46.142622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:46.142565 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:46.884463 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:46.884433 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:47.385456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.385422 4981 generic.go:356] "Generic (PLEG): container finished" podID="42a20f96-6209-41dd-add8-de189e2fba0c" containerID="f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0" exitCode=0 Nov 01 22:51:47.395892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.385501 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" event={"ID":"42a20f96-6209-41dd-add8-de189e2fba0c","Type":"ContainerDied","Data":"f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0"} Nov 01 22:51:47.395892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.386082 4981 scope.go:122] "RemoveContainer" containerID="f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0" Nov 01 22:51:47.395892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.388226 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:51:47.395892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.388293 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:51:47.395892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.388327 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:51:47.514901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.514867 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": read tcp 10.134.0.2:57786->10.134.0.220:15021: read: connection reset by peer" start-of-body= Nov 01 22:51:47.515067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.514925 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": read tcp 10.134.0.2:57786->10.134.0.220:15021: read: connection reset by peer" Nov 01 22:51:47.796816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:47.796786 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:48.409414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.409381 4981 generic.go:356] "Generic (PLEG): container finished" podID="08a8bacc-1268-463d-b491-b9a705807834" containerID="7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e" exitCode=0 Nov 01 22:51:48.409974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.409443 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" event={"ID":"08a8bacc-1268-463d-b491-b9a705807834","Type":"ContainerDied","Data":"7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e"} Nov 01 22:51:48.410372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.410351 4981 scope.go:122] "RemoveContainer" containerID="7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e" Nov 01 22:51:48.412451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.412421 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:51:48.412543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.412491 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:51:48.412543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.412525 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:51:48.413302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.413025 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" event={"ID":"42a20f96-6209-41dd-add8-de189e2fba0c","Type":"ContainerStarted","Data":"0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803"} Nov 01 22:51:48.605666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:48.605569 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:49.230975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.230941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:49.281954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.279011 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:51:49.281954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.279224 4981 patch_prober.go:29] interesting pod/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:51:49.281954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.279321 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 22:51:49.425972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.425940 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" event={"ID":"08a8bacc-1268-463d-b491-b9a705807834","Type":"ContainerStarted","Data":"1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680"} Nov 01 22:51:49.877718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.873629 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:51:49.881628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.879792 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 22:51:49.881628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.879843 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 22:51:49.965205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:49.965112 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:50.268223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:50.268188 4981 patch_prober.go:29] interesting pod/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:51:50.268427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:50.268242 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 22:51:50.549673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:50.549560 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:50.915736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:50.905874 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 22:51:50.915736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:50.905932 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 22:51:51.163972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:51.163936 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:51.271634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:51.268935 4981 patch_prober.go:29] interesting pod/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:51:51.271634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:51.269000 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 22:51:51.881601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:51.877994 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 22:51:51.881601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:51.878066 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 22:51:51.962010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:51.961975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:52.268619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:52.268562 4981 patch_prober.go:29] interesting pod/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:51:52.268875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:52.268850 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 22:51:52.704601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:52.703403 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:52.883971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:52.883933 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 22:51:52.885280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:52.885247 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 22:51:53.268658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:53.268630 4981 patch_prober.go:29] interesting pod/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 01 22:51:53.268910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:53.268885 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 01 22:51:53.437727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:53.437684 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:53.893957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:53.893921 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 22:51:53.894568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:53.894534 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 22:51:54.160438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.160346 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:54.280905 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.280874 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:51:54.283386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.283364 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:51:54.289931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.289906 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 22:51:54.707288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.707255 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924"] Nov 01 22:51:54.707517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.707490 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" containerID="cri-o://5c0f624ec8d6a3247349a01da94c76101f2e2519d82d9231b13b674e3e59feb9" gracePeriod=30 Nov 01 22:51:54.895604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.895335 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 01 22:51:54.895604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.895400 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 01 22:51:54.959660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:54.959518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:55.846459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:55.846413 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:55.878252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:55.878222 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:51:55.878700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:55.878678 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:51:55.879371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:55.879353 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:51:56.464178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:56.464144 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:57.157242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:57.157201 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:57.775702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:57.775670 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:58.373150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:58.373118 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:59.272747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:59.272716 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:51:59.939275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:51:59.939242 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:00.087246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.087215 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:52:00.123757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.123724 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7"] Nov 01 22:52:00.124787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.124769 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" Nov 01 22:52:00.124876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.124788 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" Nov 01 22:52:00.125008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.124988 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" containerName="istio-proxy" Nov 01 22:52:00.129174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.129151 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.145448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.145416 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7"] Nov 01 22:52:00.184412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.184379 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-envoy\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.184608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.184486 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istiod-ca-cert\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.184608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.184568 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-kube-api-access-8pkwt\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-kube-api-access-8pkwt\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.184741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.184634 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-credential-socket\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.185069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185038 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-credential-socket" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.185216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185066 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istiod-ca-cert" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:00.185216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185166 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-podinfo\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.185418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185399 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-socket\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.185515 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185454 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-token\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-token\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.185621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185532 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-certs\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.185621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185566 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-data\") pod \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\" (UID: \"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea\") " Nov 01 22:52:00.185901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.185876 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-socket" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.186047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.186022 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-certs" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.186186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.186165 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-data" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.187264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.187239 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.187378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.187269 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.187378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.187284 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.187378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.187298 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.187378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.187312 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.188210 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.188177 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-envoy" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:00.188408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.188383 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-podinfo" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:00.188484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.188411 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-token" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:00.189956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.189936 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-kube-api-access-8pkwt" pod "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" (UID: "6539b0b6-bb21-4d03-b7c1-c1792fcf8fea"). InnerVolumeSpecName "kube-api-access-8pkwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:00.289365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.289278 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.289897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.289412 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.289897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.289491 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.289897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.289736 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwj4\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-kube-api-access-xpwj4\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.289897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.289784 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.290149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.289898 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.290149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.289956 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.290149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.290014 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.290149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.290125 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.290364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.290328 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.290364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.290348 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.290444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.290365 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkwt\" (UniqueName: \"kubernetes.io/projected/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-kube-api-access-8pkwt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.290444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.290379 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:00.392501 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.392461 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwj4\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-kube-api-access-xpwj4\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.392741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.392518 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.392741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.392595 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.392741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.392653 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.392741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.392693 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.392972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.392768 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.392975 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.393028 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.393055 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.393098 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.393163 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.393298 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.393430 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.393613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.393477 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.395114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.395092 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.395280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.395254 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.400241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.400221 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.400456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.400439 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwj4\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-kube-api-access-xpwj4\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.458767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.458732 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:00.589923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.589834 4981 scope.go:122] "RemoveContainer" containerID="89c6221cd614baf63b15ef8aaca1f8162ad8e0b6eb93383462a9a317fb180f1e" Nov 01 22:52:00.603262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.602978 4981 scope.go:122] "RemoveContainer" containerID="dbdbf6ccc345a87c35407fa706b86fbcefb80496b1f7960738ba7636b8516517" Nov 01 22:52:00.616214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.616186 4981 scope.go:122] "RemoveContainer" containerID="5c0f624ec8d6a3247349a01da94c76101f2e2519d82d9231b13b674e3e59feb9" Nov 01 22:52:00.627824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.627805 4981 scope.go:122] "RemoveContainer" containerID="666b723a8b260ea71542f399f395fd25688813e720ea8525a5d45e8aa07f3310" Nov 01 22:52:00.634259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.634238 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" Nov 01 22:52:00.634402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.634331 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" event={"ID":"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea","Type":"ContainerDied","Data":"5c0f624ec8d6a3247349a01da94c76101f2e2519d82d9231b13b674e3e59feb9"} Nov 01 22:52:00.634402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.634387 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924" event={"ID":"6539b0b6-bb21-4d03-b7c1-c1792fcf8fea","Type":"ContainerDied","Data":"5d0df423eb1c5c774bc82d34123f02f56b5217b89b7c79fea7a45c0baeee5165"} Nov 01 22:52:00.671873 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.671839 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924"] Nov 01 22:52:00.672570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.672523 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:00.678996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.678970 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-4p924"] Nov 01 22:52:00.741968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.741932 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6539b0b6-bb21-4d03-b7c1-c1792fcf8fea" path="/var/lib/kubelet/pods/6539b0b6-bb21-4d03-b7c1-c1792fcf8fea/volumes" Nov 01 22:52:00.759568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.759542 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7"] Nov 01 22:52:00.766270 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:52:00.766236 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0e41e5_83ae_42ed_8e29_fb1a3f1c1a40.slice/crio-1613394276c2f8e2043207a69fa1c6818a98163bd9b2b650c1c40cd338b09925 WatchSource:0}: Error finding container 1613394276c2f8e2043207a69fa1c6818a98163bd9b2b650c1c40cd338b09925: Status 404 returned error can't find the container with id 1613394276c2f8e2043207a69fa1c6818a98163bd9b2b650c1c40cd338b09925 Nov 01 22:52:00.770603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.769998 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:52:00.770603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.770093 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:52:00.770603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:00.770136 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:52:01.599961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:01.599921 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:01.728373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:01.727232 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" event={"ID":"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40","Type":"ContainerStarted","Data":"157002ec4862ce6d58b450b996ff8e9a4eafe80f72408c52f3a6f2871e390e0b"} Nov 01 22:52:01.728373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:01.727332 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" event={"ID":"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40","Type":"ContainerStarted","Data":"1613394276c2f8e2043207a69fa1c6818a98163bd9b2b650c1c40cd338b09925"} Nov 01 22:52:01.758365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:01.758313 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" podStartSLOduration=83.758293718 podStartE2EDuration="1m23.758293718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:01.751253311 +0000 UTC m=+1687.179131243" watchObservedRunningTime="2026-11-01 22:52:01.758293718 +0000 UTC m=+1687.186171652" Nov 01 22:52:02.346603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:02.346558 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:02.459386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:02.459354 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:02.461836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:02.461797 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:52:02.462019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:02.461870 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 22:52:03.221647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:03.221611 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:03.459326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:03.459276 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:52:03.459560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:03.459379 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 22:52:04.344191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:04.344153 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:04.459838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:04.459800 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:52:04.460058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:04.459862 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 22:52:05.249415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:05.249384 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:05.459349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:05.459310 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:52:05.459756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:05.459394 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 22:52:06.459259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:06.459222 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 22:52:06.459476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:06.459279 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 22:52:06.519175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:06.519145 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:07.189474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:07.189440 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:07.273771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:07.273721 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:07.506355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:07.506261 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:07.646557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:07.646513 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:07.647713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:07.647682 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:07.649361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:07.649337 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:52:08.550488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:08.550455 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:09.368941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:09.368905 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:10.295056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:10.295016 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:10.305350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:10.305323 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz"] Nov 01 22:52:10.305568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:10.305547 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" containerID="cri-o://f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435" gracePeriod=30 Nov 01 22:52:11.153137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:11.153100 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:11.366796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:11.366762 4981 patch_prober.go:29] interesting pod/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:52:11.367384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:11.367345 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:52:12.081614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:12.081547 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:12.814296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:12.814191 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:13.436694 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:13.436660 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:14.268692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:14.268659 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:14.372067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:14.372034 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:14.529047 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:52:14.528954 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:52:15.001931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.001896 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:15.690448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.690423 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:52:15.733425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.733391 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z"] Nov 01 22:52:15.734781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.734754 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" Nov 01 22:52:15.734781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.734778 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" Nov 01 22:52:15.734972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.734804 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" Nov 01 22:52:15.734972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.734813 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" Nov 01 22:52:15.735123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.735109 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" Nov 01 22:52:15.735182 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.735129 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6182a7fa-5369-41de-9122-45ca7836627d" containerName="istio-proxy" Nov 01 22:52:15.740203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.740173 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.747103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.747079 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c49240-serving-cert\"" Nov 01 22:52:15.755875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.755846 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z"] Nov 01 22:52:15.842129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.841771 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-kube-api-access-jk66t\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-kube-api-access-jk66t\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.846784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.846743 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6182a7fa-5369-41de-9122-45ca7836627d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6182a7fa-5369-41de-9122-45ca7836627d-istiod-ca-cert\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.847742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.847316 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6182a7fa-5369-41de-9122-45ca7836627d-istiod-ca-cert" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:52:15.847742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.847504 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-data\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.847742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.847666 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-istio-token\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-istio-token\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.847742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.847726 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-certs\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.848249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.847943 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6182a7fa-5369-41de-9122-45ca7836627d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6182a7fa-5369-41de-9122-45ca7836627d-istio-podinfo\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.848249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.848046 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-socket\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.848249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.848074 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-envoy\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.848249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.848108 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-credential-socket\") pod \"6182a7fa-5369-41de-9122-45ca7836627d\" (UID: \"6182a7fa-5369-41de-9122-45ca7836627d\") " Nov 01 22:52:15.848793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.848770 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-data" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.849168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.849145 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-socket" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.849408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.849385 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-certs" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.849528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.849501 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-credential-socket" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.852545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.852510 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-kube-api-access-jk66t" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "kube-api-access-jk66t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:15.852721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.852555 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-istio-token" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:52:15.853024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.853001 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-envoy" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:52:15.854109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.854069 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6182a7fa-5369-41de-9122-45ca7836627d-istio-podinfo" pod "6182a7fa-5369-41de-9122-45ca7836627d" (UID: "6182a7fa-5369-41de-9122-45ca7836627d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:52:15.856228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.856186 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-metrics-certs\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.856420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.856400 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.856684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.856665 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhp22\" (UniqueName: \"kubernetes.io/projected/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-kube-api-access-zhp22\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.856845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.856828 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-maas-api-tls\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.857127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857110 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857131 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857145 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857161 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk66t\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-kube-api-access-jk66t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857175 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6182a7fa-5369-41de-9122-45ca7836627d-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857188 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857202 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6182a7fa-5369-41de-9122-45ca7836627d-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857216 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6182a7fa-5369-41de-9122-45ca7836627d-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.857539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.857229 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6182a7fa-5369-41de-9122-45ca7836627d-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:52:15.903478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.903393 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:15.909467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.909436 4981 generic.go:356] "Generic (PLEG): container finished" podID="6182a7fa-5369-41de-9122-45ca7836627d" containerID="f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435" exitCode=0 Nov 01 22:52:15.909654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.909488 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" Nov 01 22:52:15.909654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.909523 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" event={"ID":"6182a7fa-5369-41de-9122-45ca7836627d","Type":"ContainerDied","Data":"f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435"} Nov 01 22:52:15.909654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.909601 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz" event={"ID":"6182a7fa-5369-41de-9122-45ca7836627d","Type":"ContainerDied","Data":"c1337a4d2cfc8cb37555053bb9b0a17bf2e72cc1203a22c663b2c16151b96148"} Nov 01 22:52:15.909654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.909637 4981 scope.go:122] "RemoveContainer" containerID="f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435" Nov 01 22:52:15.927386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.927367 4981 scope.go:122] "RemoveContainer" containerID="9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6" Nov 01 22:52:15.953347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.953322 4981 scope.go:122] "RemoveContainer" containerID="f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435" Nov 01 22:52:15.954151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.954119 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz"] Nov 01 22:52:15.956325 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.956292 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-85zpz"] Nov 01 22:52:15.959833 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:52:15.959762 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435\": container with ID starting with f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435 not found: ID does not exist" containerID="f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435" Nov 01 22:52:15.959833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.959799 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435"} err="failed to get container status \"f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435\": rpc error: code = NotFound desc = could not find container \"f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435\": container with ID starting with f598de95e79545dd41f20e58e8c77e3a948d727b4fbe35a61dc0826eccdfa435 not found: ID does not exist" Nov 01 22:52:15.959833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.959827 4981 scope.go:122] "RemoveContainer" containerID="9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6" Nov 01 22:52:15.960233 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:52:15.960209 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6\": container with ID starting with 9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6 not found: ID does not exist" containerID="9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6" Nov 01 22:52:15.960282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.960244 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6"} err="failed to get container status \"9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6\": rpc error: code = NotFound desc = could not find container \"9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6\": container with ID starting with 9daf112dad4b27d2f4b90491a481505ba2fb91b50313811d8a5dde1ee9a6e2b6 not found: ID does not exist" Nov 01 22:52:15.961012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.960990 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-metrics-certs\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.961157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.961139 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.961320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.961302 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhp22\" (UniqueName: \"kubernetes.io/projected/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-kube-api-access-zhp22\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.961459 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.961439 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-maas-api-tls\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.962016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.961985 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.963997 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.963978 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-maas-api-tls\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.966401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.966379 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-metrics-certs\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:15.971896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:15.971870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhp22\" (UniqueName: \"kubernetes.io/projected/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-kube-api-access-zhp22\") pod \"maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:16.072893 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:16.072758 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:16.371230 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:52:16.370189 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f62aab_95cf_4fb9_86f9_89534b73e1b9.slice/crio-90d80d9933eca42d0b7fc31c124f2ca2b04a43e05d3c6fc0a9cd7a0dea80a0de WatchSource:0}: Error finding container 90d80d9933eca42d0b7fc31c124f2ca2b04a43e05d3c6fc0a9cd7a0dea80a0de: Status 404 returned error can't find the container with id 90d80d9933eca42d0b7fc31c124f2ca2b04a43e05d3c6fc0a9cd7a0dea80a0de Nov 01 22:52:16.371230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:16.370663 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z"] Nov 01 22:52:16.682844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:16.682808 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:16.789163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:16.789125 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6182a7fa-5369-41de-9122-45ca7836627d" path="/var/lib/kubelet/pods/6182a7fa-5369-41de-9122-45ca7836627d/volumes" Nov 01 22:52:16.921786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:16.921752 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" event={"ID":"d4f62aab-95cf-4fb9-86f9-89534b73e1b9","Type":"ContainerStarted","Data":"90d80d9933eca42d0b7fc31c124f2ca2b04a43e05d3c6fc0a9cd7a0dea80a0de"} Nov 01 22:52:17.490115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:17.489937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:17.937401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:17.937367 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" event={"ID":"d4f62aab-95cf-4fb9-86f9-89534b73e1b9","Type":"ContainerStarted","Data":"0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b"} Nov 01 22:52:17.937926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:17.937506 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:17.956603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:17.956444 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" podStartSLOduration=87.47711746 podStartE2EDuration="1m27.956426552s" totalImagesPullingTime="479.309092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:52:17.954494807 +0000 UTC m=+1703.382372739" watchObservedRunningTime="2026-11-01 22:52:17.956426552 +0000 UTC m=+1703.384304483" Nov 01 22:52:18.167793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:18.167760 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:18.774455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:18.774422 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:19.451282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:19.451249 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:20.125751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:20.125717 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:20.877056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:20.874265 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:21.699971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:21.699885 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:22.417242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:22.417208 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:23.095716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:23.095654 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:23.748292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:23.748258 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:23.953329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:23.953300 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 22:52:25.089509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:25.089470 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:26.051062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:26.051032 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:26.709878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:26.709840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:27.353338 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:27.353279 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:28.064626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:28.064593 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:28.795329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:28.795298 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:29.474937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:29.474902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:29.589563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:29.582339 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:30.355600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:30.355128 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:30.522548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:30.522513 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:31.267261 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:31.267227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:32.063965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:32.063928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:32.847751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:32.847701 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:33.890799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:33.890769 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:34.551537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:34.551481 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:35.242268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:35.242236 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:35.976546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:35.976510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:36.651363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:36.651315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:37.428950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:37.428921 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:38.168026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:38.167983 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:38.872746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:38.872695 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:39.612153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:39.612117 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:40.459824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:40.459792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:41.354866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:41.354832 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:42.162721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:42.162678 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:42.953562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:42.953522 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:43.696954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:43.696914 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:44.015156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:44.015074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:44.529202 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:52:44.529166 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:52:44.621561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:44.621530 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:45.417794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:45.417762 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:46.507916 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:46.507876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:47.558960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:47.558900 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:48.353309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:48.353280 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:49.105205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:49.105172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:49.803598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:49.802564 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:49.929778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:49.929734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:50.092560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:50.092473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:50.190020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:50.189982 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:50.657133 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:50.657091 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:50.861858 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:50.861811 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:51.295599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:51.294838 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:51.960649 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:51.960601 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:52.895770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:52.895711 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:53.262130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:53.262086 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:53.792474 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:53.792428 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:54.431933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:54.431894 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:55.051206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:55.051172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:55.150166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:55.150131 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:55.285402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:55.285370 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:55.717793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:55.717734 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb"] Nov 01 22:52:55.718288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:55.717982 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" containerID="cri-o://6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162" gracePeriod=30 Nov 01 22:52:56.056602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:56.056509 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:56.398812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:56.398722 4981 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:52:56.398986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:56.398804 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:52:56.841711 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:56.841675 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:57.578033 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:57.577978 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:58.287193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:58.287164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:58.706759 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:58.706721 4981 generic.go:356] "Generic (PLEG): container finished" podID="d827f366-8ae1-425e-98ab-40ed8943f512" containerID="ceeeba62efcebe1e993adc8be6279e3560a6247ebadf68e4071f33189dd642bf" exitCode=0 Nov 01 22:52:58.706961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:58.706793 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" event={"ID":"d827f366-8ae1-425e-98ab-40ed8943f512","Type":"ContainerDied","Data":"ceeeba62efcebe1e993adc8be6279e3560a6247ebadf68e4071f33189dd642bf"} Nov 01 22:52:58.707284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:58.707268 4981 scope.go:122] "RemoveContainer" containerID="ceeeba62efcebe1e993adc8be6279e3560a6247ebadf68e4071f33189dd642bf" Nov 01 22:52:58.709445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:58.709412 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:52:58.709592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:58.709498 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:52:58.709592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:58.709547 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:52:59.061475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:59.061390 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:52:59.718251 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:59.718209 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" event={"ID":"d827f366-8ae1-425e-98ab-40ed8943f512","Type":"ContainerStarted","Data":"e3d95c47c1ec518b4c45ced7758af6c343e81591a070b353533e85651d3d0b58"} Nov 01 22:52:59.899603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:52:59.889427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:00.380596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:00.379954 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:53:00.380596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:00.380083 4981 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:53:00.380596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:00.380139 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 22:53:00.700906 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:00.700870 4981 scope.go:122] "RemoveContainer" containerID="51c1504e85fedafcf9d9100c33b1c262a1159ab130d2fd40c028f1ac9d522877" Nov 01 22:53:00.712859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:00.712837 4981 scope.go:122] "RemoveContainer" containerID="ae2bb003b0e28f6bdbdbac7fce6e4c484ac77b8011bb7c2f3ad3eafcf46be6df" Nov 01 22:53:00.762719 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:00.762686 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:01.266799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.266439 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309296 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-envoy\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309355 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-credential-socket\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309435 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-podinfo\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309480 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-token\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-token\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309520 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c13c7e4-17a5-470b-becd-8c18896915d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c13c7e4-17a5-470b-becd-8c18896915d3-istiod-ca-cert\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309568 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-certs\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309723 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-data\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309775 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-kube-api-access-pjkbn\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-kube-api-access-pjkbn\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310277 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.309852 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-socket\") pod \"3c13c7e4-17a5-470b-becd-8c18896915d3\" (UID: \"3c13c7e4-17a5-470b-becd-8c18896915d3\") " Nov 01 22:53:01.310987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.310371 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-data" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.310987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.310566 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-socket" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.310987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.310601 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-certs" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.312247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.311644 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.312247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.311672 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.312247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.311687 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.312247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.312090 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c13c7e4-17a5-470b-becd-8c18896915d3-istiod-ca-cert" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:53:01.314598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.313058 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-kube-api-access-pjkbn" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "kube-api-access-pjkbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:01.314598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.314002 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-credential-socket" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.318629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.316526 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-envoy" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:53:01.318629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.318068 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-podinfo" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:53:01.318804 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.318734 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-token" pod "3c13c7e4-17a5-470b-becd-8c18896915d3" (UID: "3c13c7e4-17a5-470b-becd-8c18896915d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:53:01.327807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.327779 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7"] Nov 01 22:53:01.329976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.329952 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" Nov 01 22:53:01.330117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.330104 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" Nov 01 22:53:01.330231 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.330221 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" Nov 01 22:53:01.330311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.330301 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" Nov 01 22:53:01.330709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.330693 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" Nov 01 22:53:01.330832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.330821 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerName="istio-proxy" Nov 01 22:53:01.336433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.336286 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7"] Nov 01 22:53:01.336611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.336413 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.341059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.341029 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-64b196-openshift-default-dockercfg-h5sfj\"" Nov 01 22:53:01.400302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.399383 4981 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:53:01.400302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.399461 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 22:53:01.419966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.419933 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.420165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.419984 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c13c7e4-17a5-470b-becd-8c18896915d3-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.420165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.420004 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.420165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.420017 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.420165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.420029 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c13c7e4-17a5-470b-becd-8c18896915d3-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.420165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.420043 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkbn\" (UniqueName: \"kubernetes.io/projected/3c13c7e4-17a5-470b-becd-8c18896915d3-kube-api-access-pjkbn\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:53:01.522539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.521926 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t84tc\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-kube-api-access-t84tc\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.522539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522192 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.522539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522241 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8220b2c2-c473-4e28-9caf-e20b3fd61537-istiod-ca-cert\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.522539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522347 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-credential-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.522539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522509 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-podinfo\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.522977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522679 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-data\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.522977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522736 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-envoy\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.522977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522820 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-certs\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.523147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.522992 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-token\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.555666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.555622 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.625498 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t84tc\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-kube-api-access-t84tc\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.625738 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.625775 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8220b2c2-c473-4e28-9caf-e20b3fd61537-istiod-ca-cert\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.625806 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-credential-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.625924 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-podinfo\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.626077 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-data\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.626135 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-envoy\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.626180 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-certs\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.626416 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-token\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.628823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.627481 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.631017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.629975 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-data\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.631017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.630510 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-credential-socket\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.631828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.631806 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-certs\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.634957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.632275 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8220b2c2-c473-4e28-9caf-e20b3fd61537-istiod-ca-cert\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.640601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.640104 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-envoy\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.648138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.645484 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-podinfo\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.648138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.647730 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-token\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.650462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.648795 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t84tc\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-kube-api-access-t84tc\") pod \"e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.719638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.719606 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:01.744230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.744195 4981 generic.go:356] "Generic (PLEG): container finished" podID="3c13c7e4-17a5-470b-becd-8c18896915d3" containerID="6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162" exitCode=0 Nov 01 22:53:01.744425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.744252 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" Nov 01 22:53:01.744425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.744275 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" event={"ID":"3c13c7e4-17a5-470b-becd-8c18896915d3","Type":"ContainerDied","Data":"6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162"} Nov 01 22:53:01.744425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.744334 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb" event={"ID":"3c13c7e4-17a5-470b-becd-8c18896915d3","Type":"ContainerDied","Data":"646901fa50f1a0658182a2420f60fa1f4a8949d49586e3ab08e355859357f91c"} Nov 01 22:53:01.744425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.744363 4981 scope.go:122] "RemoveContainer" containerID="6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162" Nov 01 22:53:01.762674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.762654 4981 scope.go:122] "RemoveContainer" containerID="7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008" Nov 01 22:53:01.801484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.800917 4981 scope.go:122] "RemoveContainer" containerID="6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162" Nov 01 22:53:01.801484 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:53:01.801473 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162\": container with ID starting with 6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162 not found: ID does not exist" containerID="6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162" Nov 01 22:53:01.802088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.801499 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162"} err="failed to get container status \"6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162\": rpc error: code = NotFound desc = could not find container \"6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162\": container with ID starting with 6cbb387e2c93e2e9808e36023155219a67393c254de0dccfa1111e126f7d5162 not found: ID does not exist" Nov 01 22:53:01.802088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.801523 4981 scope.go:122] "RemoveContainer" containerID="7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008" Nov 01 22:53:01.802088 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:53:01.801819 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008\": container with ID starting with 7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008 not found: ID does not exist" containerID="7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008" Nov 01 22:53:01.802088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.801846 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008"} err="failed to get container status \"7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008\": rpc error: code = NotFound desc = could not find container \"7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008\": container with ID starting with 7c73f1d9d939ecf7604f75d1e50182fe9b05c5625a6c5e7eeccf2bf1a99cf008 not found: ID does not exist" Nov 01 22:53:01.832845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.832808 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb"] Nov 01 22:53:01.845775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:01.842070 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-5fvmb"] Nov 01 22:53:02.043478 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:53:02.043447 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8220b2c2_c473_4e28_9caf_e20b3fd61537.slice/crio-f6c00e66468bd7d48219ebd990381de2f2a41e2d14e9ad5d92907c2c6b258a76 WatchSource:0}: Error finding container f6c00e66468bd7d48219ebd990381de2f2a41e2d14e9ad5d92907c2c6b258a76: Status 404 returned error can't find the container with id f6c00e66468bd7d48219ebd990381de2f2a41e2d14e9ad5d92907c2c6b258a76 Nov 01 22:53:02.045825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.045800 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7"] Nov 01 22:53:02.046817 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.046785 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:53:02.046876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.046857 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:53:02.046922 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.046886 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:53:02.342932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.342840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:02.391596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.382034 4981 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:53:02.391596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.382100 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 22:53:02.756275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.756243 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c13c7e4-17a5-470b-becd-8c18896915d3" path="/var/lib/kubelet/pods/3c13c7e4-17a5-470b-becd-8c18896915d3/volumes" Nov 01 22:53:02.761505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.761465 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" event={"ID":"8220b2c2-c473-4e28-9caf-e20b3fd61537","Type":"ContainerStarted","Data":"80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587"} Nov 01 22:53:02.761762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.761534 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" event={"ID":"8220b2c2-c473-4e28-9caf-e20b3fd61537","Type":"ContainerStarted","Data":"f6c00e66468bd7d48219ebd990381de2f2a41e2d14e9ad5d92907c2c6b258a76"} Nov 01 22:53:02.805109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:02.805044 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" podStartSLOduration=12.805025504 podStartE2EDuration="12.805025504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:02.796256227 +0000 UTC m=+1748.224134160" watchObservedRunningTime="2026-11-01 22:53:02.805025504 +0000 UTC m=+1748.232903436" Nov 01 22:53:03.159743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.159651 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:03.389060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.389019 4981 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:53:03.389398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.389356 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 22:53:03.720135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.720096 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:03.727108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.727078 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:03.772096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.772060 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:03.773092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.773070 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:53:03.876594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:03.873073 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:04.383963 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:04.383918 4981 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 01 22:53:04.384281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:04.384250 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 01 22:53:04.600191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:04.600163 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:04.940797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:04.940762 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:05.133440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:05.133406 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:05.281357 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:05.281267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:05.380384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:05.380352 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:53:05.380895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:05.380871 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:53:05.381462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:05.381444 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 22:53:05.591975 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:05.591895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:06.357943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:06.357891 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:06.489952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:06.489917 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:07.277683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:07.277645 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:08.088493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:08.088453 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:08.202363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:08.202330 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:08.882790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:08.882753 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:09.588857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:09.588823 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:10.263472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:10.263440 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:10.953409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:10.953376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:11.763213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:11.763182 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:12.475532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:12.475496 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:13.143644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:13.143560 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:13.862417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:13.862371 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:14.345786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:14.345750 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:14.527729 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:53:14.527691 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:53:14.557781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:14.557733 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:15.185289 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:15.185259 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:16.062710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:16.062667 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:16.745704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:16.745671 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:17.548408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:17.548375 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:18.258969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:18.258938 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:18.961390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:18.961356 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:19.840969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:19.840930 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:20.552524 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:20.552492 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:21.248286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:21.248240 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:21.873842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:21.873807 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:22.567642 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:22.567598 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:23.177057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:23.177025 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:23.906622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:23.906567 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:24.482091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:24.482055 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:25.087321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:25.087290 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:25.733352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:25.733318 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:26.512283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:26.512247 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:26.625114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:26.625073 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:27.353733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:27.353699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:28.059782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:28.059745 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:28.854814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:28.854775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:29.472388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:29.472349 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:30.238607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:30.236208 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:30.847985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:30.847946 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:31.447988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:31.447947 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:32.066740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:32.066687 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:32.763110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:32.760591 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:33.271917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:33.271876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:33.984454 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:33.984415 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:34.652725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:34.652679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:35.301594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:35.301544 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:35.973889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:35.973851 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:36.787130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:36.787094 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:36.898203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:36.898119 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:37.600453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:37.600388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:38.288520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:38.288484 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:38.911494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:38.911463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:39.670456 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:39.670408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:40.491130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:40.491097 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:41.154995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:41.154930 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:41.790045 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:41.790014 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:42.628306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:42.628276 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:43.437351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:43.437323 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:44.541551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:44.541426 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:44.662312 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:53:44.662246 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:53:45.678638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:45.678606 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:46.460147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:46.460114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:47.249675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:47.249641 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:48.021343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:48.021306 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:48.818339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:48.818309 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:49.424350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:49.424319 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:50.025328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:50.025292 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr"] Nov 01 22:53:50.030464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:50.030434 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d7047c2-x6ntr"] Nov 01 22:53:50.182790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:50.182746 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:50.739764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:50.739731 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26bd4957-9025-48d7-adde-2d015113976c" path="/var/lib/kubelet/pods/26bd4957-9025-48d7-adde-2d015113976c/volumes" Nov 01 22:53:50.839207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:50.839175 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:51.444623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:51.444571 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:52.068031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:52.067987 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:52.722010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:52.721979 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:53.356781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:53.356749 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:54.084534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:54.084495 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:54.747859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:54.747826 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:55.546807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:55.546764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:56.170772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:56.170741 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:56.860761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:56.860702 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:57.625159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:57.625128 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:58.375327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:58.375293 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:59.048008 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:59.047974 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:53:59.795108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:53:59.795067 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:00.381434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:00.381390 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:00.890657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:00.890625 4981 scope.go:122] "RemoveContainer" containerID="9984df973612e97eb62d45da6ad821d2bf34ebfdc7de32b8539c0860980adb6c" Nov 01 22:54:01.143635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:01.143536 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:01.984322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:01.984283 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:02.794770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:02.794734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:03.473288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:03.473253 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:03.970177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:03.970129 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:04.076881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:04.076838 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:04.251932 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:04.251831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:04.879366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:04.879250 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:05.560005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:05.559969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:06.217761 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:06.217726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:06.952599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:06.952557 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:07.673551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:07.673510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:08.341928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:08.341891 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:09.187652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:09.182644 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:09.750376 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:09.750325 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:10.531555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:10.531526 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:11.229511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:11.229474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:11.980021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:11.979989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:12.662743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:12.662700 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:13.362554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:13.362516 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:13.975306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:13.975263 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:14.392104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:14.392025 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:14.529015 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:54:14.528982 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:54:14.812600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:14.812528 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:15.560912 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:15.560876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:16.281825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:16.281794 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:17.174791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:17.174736 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:17.765856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:17.765814 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:18.570301 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:18.570270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:19.245180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:19.245124 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:19.886097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:19.886062 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:20.691118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:20.691083 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:20.790645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:20.790608 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:21.808170 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:21.808135 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:22.592756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:22.592726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:23.275837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:23.275802 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:24.039282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:24.039250 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:24.663350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:24.663315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:25.453787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:25.452189 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:26.095729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:26.095697 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:26.845534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:26.845501 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:27.584017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:27.583982 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:28.341723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:28.341689 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:29.152708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:29.152673 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:29.879322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:29.879286 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:30.567116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:30.567089 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:31.348012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:31.347974 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:32.071296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:32.071263 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:32.738508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:32.738470 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:33.096085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:33.096001 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:33.842917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:33.842883 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:34.849915 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:34.849872 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:35.182780 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:35.182742 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:35.283224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:35.283194 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:35.757760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:35.757717 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:36.492184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:36.492156 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:37.254272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:37.254244 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:38.082003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:38.081972 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:38.757819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:38.757784 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:39.506987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:39.506941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:40.161043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:40.161009 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:40.702051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:40.702014 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts"] Nov 01 22:54:40.702669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:40.702283 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" containerID="cri-o://a37210e2dc883f37cf8be844ebd082af064e77e6e0d90e99996b5ab96d2c9db1" gracePeriod=30 Nov 01 22:54:40.715546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:40.715512 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6"] Nov 01 22:54:40.715794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:40.715767 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" containerID="cri-o://e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5" gracePeriod=30 Nov 01 22:54:40.937343 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:40.937305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:41.661235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:41.661207 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:42.466329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:42.466293 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:43.172696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:43.172662 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:43.826762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:43.826724 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7"] Nov 01 22:54:43.900874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:43.900839 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:44.002027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:44.001993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:44.245960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:44.245924 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:44.586570 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:54:44.586484 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:54:45.634059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:45.634021 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:46.311360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.311336 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:54:46.354237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.354203 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577"] Nov 01 22:54:46.355848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.355822 4981 cpu_manager.go:414] "containerMap: removing container" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" Nov 01 22:54:46.355848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.355848 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" Nov 01 22:54:46.356150 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.356130 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" containerName="istio-proxy" Nov 01 22:54:46.360966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.360903 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365141 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64e27162-6859-45a7-a557-8e9ddd63c463-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e27162-6859-45a7-a557-8e9ddd63c463-istio-podinfo\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365202 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e27162-6859-45a7-a557-8e9ddd63c463-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e27162-6859-45a7-a557-8e9ddd63c463-istiod-ca-cert\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365326 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-kube-api-access-sk6lf\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-kube-api-access-sk6lf\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365406 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-data\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365442 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-istio-token\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-istio-token\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365479 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-credential-socket\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365567 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-certs\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365642 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-socket\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.366541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.365695 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-envoy\") pod \"64e27162-6859-45a7-a557-8e9ddd63c463\" (UID: \"64e27162-6859-45a7-a557-8e9ddd63c463\") " Nov 01 22:54:46.369171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.369146 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-envoy" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.371754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.371725 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64e27162-6859-45a7-a557-8e9ddd63c463-istio-podinfo" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:54:46.372432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.372408 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577"] Nov 01 22:54:46.372897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.372875 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-021714-openshift-default-dockercfg-vmtmc\"" Nov 01 22:54:46.374818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.374786 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-credential-socket" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.375339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.375314 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-certs" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.375534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.375515 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-socket" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.375729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.375710 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-data" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.375998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.375978 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e27162-6859-45a7-a557-8e9ddd63c463-istiod-ca-cert" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:46.379855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.379826 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-kube-api-access-sk6lf" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "kube-api-access-sk6lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:46.380938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.380913 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-istio-token" pod "64e27162-6859-45a7-a557-8e9ddd63c463" (UID: "64e27162-6859-45a7-a557-8e9ddd63c463"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:46.469781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.469746 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.469995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.469823 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.469995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.469894 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tqm\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-kube-api-access-q2tqm\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.469995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.469975 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.470180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.470168 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.470237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.470206 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.470419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.470378 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.470914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.470885 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.471036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.470950 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.473944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.473916 4981 generic.go:356] "Generic (PLEG): container finished" podID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerID="a37210e2dc883f37cf8be844ebd082af064e77e6e0d90e99996b5ab96d2c9db1" exitCode=0 Nov 01 22:54:46.474138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474115 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk6lf\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-kube-api-access-sk6lf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474145 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474161 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64e27162-6859-45a7-a557-8e9ddd63c463-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474299 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474314 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474462 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474484 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64e27162-6859-45a7-a557-8e9ddd63c463-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474524 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" event={"ID":"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a","Type":"ContainerDied","Data":"a37210e2dc883f37cf8be844ebd082af064e77e6e0d90e99996b5ab96d2c9db1"} Nov 01 22:54:46.474643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474507 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64e27162-6859-45a7-a557-8e9ddd63c463-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474648 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64e27162-6859-45a7-a557-8e9ddd63c463-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.474771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474648 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" event={"ID":"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a","Type":"ContainerDied","Data":"2901a33b2117f22571f5b4f6d93b981f33e92a41ed9024342ee0e0793f5e6efe"} Nov 01 22:54:46.474877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.474828 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2901a33b2117f22571f5b4f6d93b981f33e92a41ed9024342ee0e0793f5e6efe" Nov 01 22:54:46.486124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.486090 4981 generic.go:356] "Generic (PLEG): container finished" podID="64e27162-6859-45a7-a557-8e9ddd63c463" containerID="e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5" exitCode=0 Nov 01 22:54:46.486368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.486345 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" event={"ID":"64e27162-6859-45a7-a557-8e9ddd63c463","Type":"ContainerDied","Data":"e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5"} Nov 01 22:54:46.486470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.486413 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" event={"ID":"64e27162-6859-45a7-a557-8e9ddd63c463","Type":"ContainerDied","Data":"930b1c585b32c00cab0ca72cd61b53cd55b61ddd263e2520a032ceb83a96ae0e"} Nov 01 22:54:46.486470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.486454 4981 scope.go:122] "RemoveContainer" containerID="e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5" Nov 01 22:54:46.486702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.486685 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6" Nov 01 22:54:46.493634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.493614 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:54:46.506299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.506281 4981 scope.go:122] "RemoveContainer" containerID="e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5" Nov 01 22:54:46.506612 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:54:46.506565 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5\": container with ID starting with e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5 not found: ID does not exist" containerID="e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5" Nov 01 22:54:46.506662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.506604 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5"} err="failed to get container status \"e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5\": rpc error: code = NotFound desc = could not find container \"e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5\": container with ID starting with e6eefb3208da1d649d5adc0ababbd07f7f33484fb26a02aaa68ec9526e07a4b5 not found: ID does not exist" Nov 01 22:54:46.536461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.536422 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl"] Nov 01 22:54:46.547402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.547290 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" Nov 01 22:54:46.559355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.559325 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" Nov 01 22:54:46.559994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.559971 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" Nov 01 22:54:46.566570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.566546 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6"] Nov 01 22:54:46.566702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.566621 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl"] Nov 01 22:54:46.566702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.566661 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-ldqj6"] Nov 01 22:54:46.566702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.566678 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.575283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575256 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-envoy\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.575422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575319 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istiod-ca-cert\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.575422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575387 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-socket\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.575631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575510 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-token\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-token\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.575717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575640 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-data\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.575717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575682 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-kube-api-access-k96hc\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-kube-api-access-k96hc\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.575845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575709 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-socket" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.575845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575772 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-certs\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.575845 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575773 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istiod-ca-cert" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:54:46.576028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575847 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-credential-socket\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.576028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575889 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-podinfo\") pod \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\" (UID: \"8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a\") " Nov 01 22:54:46.576028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.575934 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-data" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.576329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576200 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-certs" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.576329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576305 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.576457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576366 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.576632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576593 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.576798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576659 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.576798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576709 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tqm\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-kube-api-access-q2tqm\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.576798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576760 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.576798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576776 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.577069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576851 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.577069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576884 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.577069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.576940 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.577238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.577221 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.577314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.577243 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.577314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.577258 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.577314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.577272 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.577790 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.577759 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.577983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.577955 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-envoy" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.578071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.578015 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.578201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.578175 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-credential-socket" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:54:46.578275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.578243 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.578336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.578301 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.579687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.579644 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-kube-api-access-k96hc" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "kube-api-access-k96hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:46.580984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.580962 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.580984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.580970 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-token" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:54:46.581147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.581128 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.581395 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.581376 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-podinfo" pod "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" (UID: "8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:54:46.586807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.586744 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tqm\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-kube-api-access-q2tqm\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.587712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.587689 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-cw577\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.590784 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.590755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679147 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679207 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679241 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679302 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679359 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh77z\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-kube-api-access-lh77z\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679411 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679527 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679742 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679807 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679934 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k96hc\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-kube-api-access-k96hc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679954 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679970 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679985 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.680597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.679999 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:54:46.697435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.697395 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:46.747651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.747549 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e27162-6859-45a7-a557-8e9ddd63c463" path="/var/lib/kubelet/pods/64e27162-6859-45a7-a557-8e9ddd63c463/volumes" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781172 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh77z\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-kube-api-access-lh77z\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781225 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781338 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781481 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781519 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781754 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781778 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781799 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.781860 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.782635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.782246 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.783304 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.782765 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.792650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.784785 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.792650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.786126 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.792650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.786431 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.792650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.789387 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.796936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.794710 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh77z\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-kube-api-access-lh77z\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.796936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.796439 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.798447 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.798399 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:46.895549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:46.895492 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:47.052124 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:54:47.052089 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda949c1c2_3db6_427c_a5cf_29824eb9d3d6.slice/crio-d81f877049779f0935fc0d85a00ccd4d59290cf6ee8162fca8f2c8d5946fa3a7 WatchSource:0}: Error finding container d81f877049779f0935fc0d85a00ccd4d59290cf6ee8162fca8f2c8d5946fa3a7: Status 404 returned error can't find the container with id d81f877049779f0935fc0d85a00ccd4d59290cf6ee8162fca8f2c8d5946fa3a7 Nov 01 22:54:47.052341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.052278 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577"] Nov 01 22:54:47.057140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.057106 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:47.057272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.057193 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:47.057272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.057243 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:47.233753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.233704 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:47.234021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.233997 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:47.234186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.234168 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:47.235995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.235974 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl"] Nov 01 22:54:47.297672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.294461 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:47.441201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.441164 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:54:47.441337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.441279 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:54:47.545800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.542952 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" event={"ID":"a949c1c2-3db6-427c-a5cf-29824eb9d3d6","Type":"ContainerStarted","Data":"3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec"} Nov 01 22:54:47.545800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.543016 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" event={"ID":"a949c1c2-3db6-427c-a5cf-29824eb9d3d6","Type":"ContainerStarted","Data":"d81f877049779f0935fc0d85a00ccd4d59290cf6ee8162fca8f2c8d5946fa3a7"} Nov 01 22:54:47.558558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.547319 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" event={"ID":"b7f690d3-a45f-4ebf-b366-24bd677fe19e","Type":"ContainerStarted","Data":"f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1"} Nov 01 22:54:47.558558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.547397 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" event={"ID":"b7f690d3-a45f-4ebf-b366-24bd677fe19e","Type":"ContainerStarted","Data":"33fdc08c3b66744dca287926ca06645218e80da39448722c23349c2d3c1b2872"} Nov 01 22:54:47.558558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.550016 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts" Nov 01 22:54:47.627019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.626940 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podStartSLOduration=21.626920769 podStartE2EDuration="21.626920769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:47.571536447 +0000 UTC m=+1852.999414381" watchObservedRunningTime="2026-11-01 22:54:47.626920769 +0000 UTC m=+1853.054798703" Nov 01 22:54:47.628353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.628292 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podStartSLOduration=21.628278867 podStartE2EDuration="21.628278867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:47.621228235 +0000 UTC m=+1853.049106169" watchObservedRunningTime="2026-11-01 22:54:47.628278867 +0000 UTC m=+1853.056156799" Nov 01 22:54:47.646020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.645993 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts"] Nov 01 22:54:47.650896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.650868 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-qdgts"] Nov 01 22:54:47.708752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.708665 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:47.907774 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:47.907647 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:48.043568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:48.043521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:48.720599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:48.720150 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 22:54:48.720599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:48.720239 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 22:54:48.891603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:48.870129 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:48.891603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:48.875971 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" podUID="a8cabded-6252-4cb6-95c8-f0ec9024a589" containerName="payload-processing" containerID="cri-o://a183bfad774e2a93eb57f11966efdeff1fbeec862f7687c52524f0ef44df5a49" gracePeriod=25 Nov 01 22:54:48.974683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:48.951926 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 22:54:48.974683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:48.952016 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 22:54:49.071526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.071216 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a" path="/var/lib/kubelet/pods/8e5fce05-fe5b-4fc7-9acb-75f7a7e65a7a/volumes" Nov 01 22:54:49.322651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.313176 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:49.500597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.493213 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:49.736597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.735550 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 22:54:49.736597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.735629 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 22:54:49.926246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.847996 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:49.928154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.926604 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 22:54:49.928405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:49.928377 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 22:54:50.087669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.065948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:50.369972 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.369863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:50.547986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.536726 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 22:54:50.649658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.648837 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:54:50.652602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.650168 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 01 22:54:50.700706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.700673 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 22:54:50.700978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.700948 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 22:54:50.756058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.756002 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:50.910659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.908756 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 22:54:50.910659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:50.908813 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 22:54:51.568189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:51.568146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:51.700613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:51.699262 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 22:54:51.700613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:51.699327 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 22:54:51.864663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:51.862238 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:51.910600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:51.909960 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 22:54:51.910600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:51.910018 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 22:54:51.959306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:51.959267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:52.366370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:52.366335 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:52.713603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:52.708122 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 22:54:52.713603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:52.708195 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 22:54:52.961753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:52.961707 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 22:54:52.962267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:52.961762 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 22:54:53.055688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:53.055220 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:53.721600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:53.716233 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 22:54:53.721600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:53.716306 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 22:54:53.816332 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:53.816180 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:53.911600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:53.909024 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 22:54:53.911600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:53.909088 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 22:54:54.495493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:54.495436 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" podUID="a8cabded-6252-4cb6-95c8-f0ec9024a589" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:54:54.651885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:54.651847 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:54.905751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:54.905671 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 01 22:54:54.905751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:54.905735 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 01 22:54:54.915036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:54.915000 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:54.915825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:54.915805 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:54.916709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:54.916691 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 22:54:55.698689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:55.698648 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:55.911151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:55.911115 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:55.911645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:55.911605 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:55.912673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:55.912643 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 22:54:56.472097 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:56.472068 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:57.286669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.286636 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:57.558149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.558074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:57.676637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.676548 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:57.864444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.864360 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6"] Nov 01 22:54:57.864718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.864688 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" containerID="cri-o://275819ea563b9e380c0216b96bca273602e0c875cf547863a4f19a91f6b5f583" gracePeriod=30 Nov 01 22:54:57.906491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.906454 4981 patch_prober.go:29] interesting pod/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:54:57.906798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.906520 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:54:57.907876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:57.907828 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:58.537244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:58.537202 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:59.224721 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.224690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:54:59.239171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.239141 4981 generic.go:356] "Generic (PLEG): container finished" podID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerID="80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587" exitCode=0 Nov 01 22:54:59.239348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.239212 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" event={"ID":"8220b2c2-c473-4e28-9caf-e20b3fd61537","Type":"ContainerDied","Data":"80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587"} Nov 01 22:54:59.239715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.239698 4981 scope.go:122] "RemoveContainer" containerID="80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587" Nov 01 22:54:59.241799 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.241773 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:59.241881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.241845 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:59.241924 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.241887 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:54:59.957271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:54:59.957232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:00.250317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.250224 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" event={"ID":"8220b2c2-c473-4e28-9caf-e20b3fd61537","Type":"ContainerStarted","Data":"1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3"} Nov 01 22:55:00.628855 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.628718 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:00.726724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.725940 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:55:00.741599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.736760 4981 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:55:00.741599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.736827 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 22:55:00.930957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.930926 4981 scope.go:122] "RemoveContainer" containerID="8429056fc870c1bfe0fcecb6a71986b54e0b6005308b1fb61aeb241ff06df78f" Nov 01 22:55:00.943807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.943786 4981 scope.go:122] "RemoveContainer" containerID="edb2a29755adfbf3f97bfcd4f0f9d81324ceeabc6e33c89f9315d4d5ca2d5254" Nov 01 22:55:00.954216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.954170 4981 scope.go:122] "RemoveContainer" containerID="a37210e2dc883f37cf8be844ebd082af064e77e6e0d90e99996b5ab96d2c9db1" Nov 01 22:55:00.964498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:00.964475 4981 scope.go:122] "RemoveContainer" containerID="a61199bc8774e5f7e5d7b2a57364221d26821004ab2f3c0afd2c447bdaa2e44f" Nov 01 22:55:01.243928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:01.243835 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:01.720354 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:01.720312 4981 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:55:01.720594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:01.720393 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 22:55:01.972197 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:01.972057 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:02.720264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:02.720230 4981 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:55:02.720483 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:02.720302 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 22:55:03.014608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.012928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:03.313104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.313016 4981 generic.go:356] "Generic (PLEG): container finished" podID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerID="275819ea563b9e380c0216b96bca273602e0c875cf547863a4f19a91f6b5f583" exitCode=0 Nov 01 22:55:03.313279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.313148 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" event={"ID":"bcddcb52-6b43-4652-8057-9f898efb0a41","Type":"ContainerDied","Data":"275819ea563b9e380c0216b96bca273602e0c875cf547863a4f19a91f6b5f583"} Nov 01 22:55:03.313279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.313205 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" event={"ID":"bcddcb52-6b43-4652-8057-9f898efb0a41","Type":"ContainerDied","Data":"603b6a32a9632cddc8cbe3db22678419ed9263d6b744a6c76ce860516ac816de"} Nov 01 22:55:03.313279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.313227 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="603b6a32a9632cddc8cbe3db22678419ed9263d6b744a6c76ce860516ac816de" Nov 01 22:55:03.313279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.313245 4981 scope.go:122] "RemoveContainer" containerID="fb12f37016e03f205114c0f4b7aeed2f215d69516fae46772910ca9b68562acc" Nov 01 22:55:03.316913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.316882 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:55:03.342186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.342149 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-data\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.342372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.342235 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcddcb52-6b43-4652-8057-9f898efb0a41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcddcb52-6b43-4652-8057-9f898efb0a41-istiod-ca-cert\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.342372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.342340 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-socket\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.342505 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.342471 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-podinfo\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.342654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.342633 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-credential-socket\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.342952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.342927 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-kube-api-access-csjwd\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-kube-api-access-csjwd\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.343101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.343083 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-certs\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.343412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.343356 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-token\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-token\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.344131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.344105 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-envoy\") pod \"bcddcb52-6b43-4652-8057-9f898efb0a41\" (UID: \"bcddcb52-6b43-4652-8057-9f898efb0a41\") " Nov 01 22:55:03.345230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.344948 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-certs" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.345230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.345192 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-socket" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.345230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.345208 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-data" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.345641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.345613 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.345733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.345641 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.345733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.345656 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.345733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.345719 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-credential-socket" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.345918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.345896 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcddcb52-6b43-4652-8057-9f898efb0a41-istiod-ca-cert" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:03.350684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.350640 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-token" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:03.350792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.350717 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-kube-api-access-csjwd" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "kube-api-access-csjwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:03.350951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.350931 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-envoy" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:03.355797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.355767 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-podinfo" pod "bcddcb52-6b43-4652-8057-9f898efb0a41" (UID: "bcddcb52-6b43-4652-8057-9f898efb0a41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:03.358868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.358849 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4"] Nov 01 22:55:03.360212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.360189 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" Nov 01 22:55:03.360212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.360211 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" Nov 01 22:55:03.360344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.360250 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" Nov 01 22:55:03.360344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.360257 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" Nov 01 22:55:03.360409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.360391 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" Nov 01 22:55:03.360409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.360406 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" containerName="istio-proxy" Nov 01 22:55:03.409415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.409384 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4"] Nov 01 22:55:03.409607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.409487 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.413242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.413208 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-64b196-serving-cert\"" Nov 01 22:55:03.447131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447080 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677119b8-8fdb-4607-92ff-1158dda37e4a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.447331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447178 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxdr\" (UniqueName: \"kubernetes.io/projected/677119b8-8fdb-4607-92ff-1158dda37e4a-kube-api-access-4sxdr\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.447443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447422 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-maas-api-tls\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.447533 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447518 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-metrics-certs\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.447775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447756 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.447879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447779 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.447879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447840 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcddcb52-6b43-4652-8057-9f898efb0a41-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.447879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447862 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcddcb52-6b43-4652-8057-9f898efb0a41-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.447879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447876 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcddcb52-6b43-4652-8057-9f898efb0a41-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.448094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.447891 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csjwd\" (UniqueName: \"kubernetes.io/projected/bcddcb52-6b43-4652-8057-9f898efb0a41-kube-api-access-csjwd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:03.549307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.549158 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677119b8-8fdb-4607-92ff-1158dda37e4a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.549307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.549307 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxdr\" (UniqueName: \"kubernetes.io/projected/677119b8-8fdb-4607-92ff-1158dda37e4a-kube-api-access-4sxdr\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.549592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.549480 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-maas-api-tls\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.549657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.549622 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-metrics-certs\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.549822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.549794 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677119b8-8fdb-4607-92ff-1158dda37e4a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.552453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.552426 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-metrics-certs\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.552935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.552900 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-maas-api-tls\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.561173 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.561134 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxdr\" (UniqueName: \"kubernetes.io/projected/677119b8-8fdb-4607-92ff-1158dda37e4a-kube-api-access-4sxdr\") pod \"maas-api-e2e-inf-b-64b196-6b4698b899-lstl4\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.623864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.623776 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:03.735608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.729211 4981 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:55:03.735608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.729279 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 22:55:03.735608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.730853 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:03.802617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.793212 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:55:03.849422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.848954 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr"] Nov 01 22:55:03.849422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.849156 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" containerID="cri-o://c643be0b55327451feb71ddb41d7e50f82e5db204c13b379df5f47cf6238a63e" gracePeriod=30 Nov 01 22:55:03.855866 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.855811 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg"] Nov 01 22:55:03.856092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:03.856062 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" containerID="cri-o://e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d" gracePeriod=30 Nov 01 22:55:04.196941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.173567 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4"] Nov 01 22:55:04.201609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.199906 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 22:55:04.405608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.380394 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6" Nov 01 22:55:04.406549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.406445 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:04.408716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.408640 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" event={"ID":"677119b8-8fdb-4607-92ff-1158dda37e4a","Type":"ContainerStarted","Data":"9749b35975879ea40086c5bdefc0c6ea7b7171d47ce920cb1f0bd7230e8943e4"} Nov 01 22:55:04.532691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.528919 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6"] Nov 01 22:55:04.533605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.533165 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" podUID="a8cabded-6252-4cb6-95c8-f0ec9024a589" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:55:04.544058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.542344 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1799f8-openshift-default-74996455f-r9lc6"] Nov 01 22:55:04.720901 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.720876 4981 patch_prober.go:29] interesting pod/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 22:55:04.721030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.720932 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 22:55:04.741757 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:04.741730 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcddcb52-6b43-4652-8057-9f898efb0a41" path="/var/lib/kubelet/pods/bcddcb52-6b43-4652-8057-9f898efb0a41/volumes" Nov 01 22:55:05.069290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:05.069207 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:05.421944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:05.421846 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" event={"ID":"677119b8-8fdb-4607-92ff-1158dda37e4a","Type":"ContainerStarted","Data":"fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d"} Nov 01 22:55:05.422340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:05.421964 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:05.445634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:05.445569 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" podStartSLOduration=98.934667065 podStartE2EDuration="1m39.445550377s" totalImagesPullingTime="510.883312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:05.439898905 +0000 UTC m=+1870.867776837" watchObservedRunningTime="2026-11-01 22:55:05.445550377 +0000 UTC m=+1870.873428308" Nov 01 22:55:05.727627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:05.727553 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:55:05.729135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:05.729108 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 22:55:05.875034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:05.874993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:06.664268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:06.664235 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:07.349030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:07.348945 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:08.156831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:08.156797 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:08.382229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:08.382190 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:08.498321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:08.498287 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:08.581950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:08.581903 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs"] Nov 01 22:55:08.590252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:08.590213 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj"] Nov 01 22:55:08.670638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:08.670608 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:08.811058 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:08.810964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:09.318756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.318679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:09.436662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.435114 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:55:09.465235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.465185 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:55:09.468470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.468417 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" event={"ID":"1b7cafbc-700c-43eb-871f-1427195d090d","Type":"ContainerDied","Data":"c643be0b55327451feb71ddb41d7e50f82e5db204c13b379df5f47cf6238a63e"} Nov 01 22:55:09.499132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.486621 4981 generic.go:356] "Generic (PLEG): container finished" podID="1b7cafbc-700c-43eb-871f-1427195d090d" containerID="c643be0b55327451feb71ddb41d7e50f82e5db204c13b379df5f47cf6238a63e" exitCode=0 Nov 01 22:55:09.517488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.517361 4981 generic.go:356] "Generic (PLEG): container finished" podID="a8cabded-6252-4cb6-95c8-f0ec9024a589" containerID="a183bfad774e2a93eb57f11966efdeff1fbeec862f7687c52524f0ef44df5a49" exitCode=0 Nov 01 22:55:09.517677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.517530 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" event={"ID":"a8cabded-6252-4cb6-95c8-f0ec9024a589","Type":"ContainerDied","Data":"a183bfad774e2a93eb57f11966efdeff1fbeec862f7687c52524f0ef44df5a49"} Nov 01 22:55:09.517677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.517606 4981 scope.go:122] "RemoveContainer" containerID="a183bfad774e2a93eb57f11966efdeff1fbeec862f7687c52524f0ef44df5a49" Nov 01 22:55:09.523630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.523004 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7" Nov 01 22:55:09.525722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.525305 4981 generic.go:356] "Generic (PLEG): container finished" podID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerID="e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d" exitCode=0 Nov 01 22:55:09.525722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.525411 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" event={"ID":"0369ac9e-dda1-4f8a-878a-24bddecac57b","Type":"ContainerDied","Data":"e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d"} Nov 01 22:55:09.525722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.525460 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" event={"ID":"0369ac9e-dda1-4f8a-878a-24bddecac57b","Type":"ContainerDied","Data":"42d589a0c3ff2c580038b06831f0432d13335f92b86f64d2448ee398ea4c7397"} Nov 01 22:55:09.525722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.525538 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg" Nov 01 22:55:09.535876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.535846 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn"] Nov 01 22:55:09.537709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.537683 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" Nov 01 22:55:09.537911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.537893 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" Nov 01 22:55:09.538067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.538053 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a8cabded-6252-4cb6-95c8-f0ec9024a589" containerName="payload-processing" Nov 01 22:55:09.538175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.538162 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cabded-6252-4cb6-95c8-f0ec9024a589" containerName="payload-processing" Nov 01 22:55:09.539386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.539365 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" containerName="istio-proxy" Nov 01 22:55:09.539542 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.539529 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cabded-6252-4cb6-95c8-f0ec9024a589" containerName="payload-processing" Nov 01 22:55:09.560671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.560642 4981 scope.go:122] "RemoveContainer" containerID="e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d" Nov 01 22:55:09.572679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.572559 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-credential-socket\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.572679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.572668 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-certs\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.572714 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-podinfo\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.572801 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-token\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-token\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.572930 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-data\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.573063 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0369ac9e-dda1-4f8a-878a-24bddecac57b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0369ac9e-dda1-4f8a-878a-24bddecac57b-istiod-ca-cert\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.573254 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-envoy\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.573336 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cabded-6252-4cb6-95c8-f0ec9024a589-kube-api-access-kbgj5\" (UniqueName: \"kubernetes.io/projected/a8cabded-6252-4cb6-95c8-f0ec9024a589-kube-api-access-kbgj5\") pod \"a8cabded-6252-4cb6-95c8-f0ec9024a589\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.573368 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-socket\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.573476 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-kube-api-access-nc5km\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-kube-api-access-nc5km\") pod \"0369ac9e-dda1-4f8a-878a-24bddecac57b\" (UID: \"0369ac9e-dda1-4f8a-878a-24bddecac57b\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.573508 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cabded-6252-4cb6-95c8-f0ec9024a589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cabded-6252-4cb6-95c8-f0ec9024a589-plugins-config-volume\") pod \"a8cabded-6252-4cb6-95c8-f0ec9024a589\" (UID: \"a8cabded-6252-4cb6-95c8-f0ec9024a589\") " Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.574837 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cabded-6252-4cb6-95c8-f0ec9024a589-plugins-config-volume" pod "a8cabded-6252-4cb6-95c8-f0ec9024a589" (UID: "a8cabded-6252-4cb6-95c8-f0ec9024a589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.575183 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-credential-socket" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.575416 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-certs" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.576657 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn"] Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.576822 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.579462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.578159 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-podinfo" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:09.588565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.584715 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0369ac9e-dda1-4f8a-878a-24bddecac57b-istiod-ca-cert" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:09.597120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.590754 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-socket" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.603923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.597500 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-kube-api-access-nc5km" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "kube-api-access-nc5km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:09.603923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.597546 4981 scope.go:122] "RemoveContainer" containerID="e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d" Nov 01 22:55:09.603923 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:09.599147 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d\": container with ID starting with e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d not found: ID does not exist" containerID="e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d" Nov 01 22:55:09.603923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.599188 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d"} err="failed to get container status \"e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d\": rpc error: code = NotFound desc = could not find container \"e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d\": container with ID starting with e36dd92cf9b7d2dc2ad4eaa590e2a885d06bad3fddec907b7a253cefac8f483d not found: ID does not exist" Nov 01 22:55:09.603923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.599512 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-data" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.610969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.610928 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-token" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:09.612399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.612366 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-envoy" pod "0369ac9e-dda1-4f8a-878a-24bddecac57b" (UID: "0369ac9e-dda1-4f8a-878a-24bddecac57b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.613359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.613332 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cabded-6252-4cb6-95c8-f0ec9024a589-kube-api-access-kbgj5" pod "a8cabded-6252-4cb6-95c8-f0ec9024a589" (UID: "a8cabded-6252-4cb6-95c8-f0ec9024a589"). InnerVolumeSpecName "kube-api-access-kbgj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:09.676440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.676394 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c2c33-56f7-45c5-911d-0413c08efb63-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.676663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.676528 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbj8\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-kube-api-access-dfbj8\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.676663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.676650 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.676810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.676770 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.676872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.676806 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.676935 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.676921 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.677006 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.676985 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.677072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677020 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.677153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677137 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.677394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677374 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc5km\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-kube-api-access-nc5km\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677400 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8cabded-6252-4cb6-95c8-f0ec9024a589-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677417 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677433 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677447 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677462 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677477 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677491 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0369ac9e-dda1-4f8a-878a-24bddecac57b-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677505 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677519 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbgj5\" (UniqueName: \"kubernetes.io/projected/a8cabded-6252-4cb6-95c8-f0ec9024a589-kube-api-access-kbgj5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.677814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.677533 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0369ac9e-dda1-4f8a-878a-24bddecac57b-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.721399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.721370 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:55:09.754785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.754747 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd"] Nov 01 22:55:09.755944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.755923 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" Nov 01 22:55:09.756037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.755944 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" Nov 01 22:55:09.756137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.756127 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" containerName="istio-proxy" Nov 01 22:55:09.760267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.760241 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.771305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.771276 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd"] Nov 01 22:55:09.780744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.780714 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-data\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.780911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.780852 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-envoy\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.780911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.780894 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-istio-token\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-istio-token\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.781073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781058 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b7cafbc-700c-43eb-871f-1427195d090d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7cafbc-700c-43eb-871f-1427195d090d-istio-podinfo\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.781138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781113 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-certs\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.781195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781165 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-credential-socket\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.781257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781212 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-socket\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.781368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781352 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7cafbc-700c-43eb-871f-1427195d090d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7cafbc-700c-43eb-871f-1427195d090d-istiod-ca-cert\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.781425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781405 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-kube-api-access-78zks\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-kube-api-access-78zks\") pod \"1b7cafbc-700c-43eb-871f-1427195d090d\" (UID: \"1b7cafbc-700c-43eb-871f-1427195d090d\") " Nov 01 22:55:09.781802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781779 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c2c33-56f7-45c5-911d-0413c08efb63-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.781920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781861 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbj8\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-kube-api-access-dfbj8\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.781920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.781896 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.782036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.782021 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.782092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.782054 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.782191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.782176 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.782267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.782252 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.782331 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.782283 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.782422 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.782408 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.782829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.782809 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.783022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.783001 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-data" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.784286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.783865 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.784286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.783899 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.784286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.783970 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c2c33-56f7-45c5-911d-0413c08efb63-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.784286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.784110 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.784286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.784213 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-socket" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.784760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.784739 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7cafbc-700c-43eb-871f-1427195d090d-istiod-ca-cert" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:09.784989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.784972 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-certs" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.785375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.785341 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-credential-socket" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.787914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.787894 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b7cafbc-700c-43eb-871f-1427195d090d-istio-podinfo" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:09.791660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.791636 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbj8\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-kube-api-access-dfbj8\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.792475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.792452 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.794861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.794819 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.795089 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.795063 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-kube-api-access-78zks" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "kube-api-access-78zks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:09.795190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.795133 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.795309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.795287 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-envoy" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:09.796643 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.796622 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-istio-token" pod "1b7cafbc-700c-43eb-871f-1427195d090d" (UID: "1b7cafbc-700c-43eb-871f-1427195d090d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:09.884747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.884689 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh97q\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-kube-api-access-gh97q\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.884871 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.884922 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885023 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885053 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58874768-db63-47ff-b564-ff6c26f84b86-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885088 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885142 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58874768-db63-47ff-b564-ff6c26f84b86-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885174 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885253 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885371 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7cafbc-700c-43eb-871f-1427195d090d-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885384 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78zks\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-kube-api-access-78zks\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885401 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885415 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885428 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7cafbc-700c-43eb-871f-1427195d090d-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885438 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7cafbc-700c-43eb-871f-1427195d090d-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885449 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885457 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.885548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.885469 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7cafbc-700c-43eb-871f-1427195d090d-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:09.911704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.911666 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:09.922943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.922916 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg"] Nov 01 22:55:09.928477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.928449 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gr8cg"] Nov 01 22:55:09.943830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.943792 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7"] Nov 01 22:55:09.949722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.949696 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1799f8-869c554fd9-8xvr7"] Nov 01 22:55:09.987716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987103 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.987716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987148 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58874768-db63-47ff-b564-ff6c26f84b86-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.987716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987177 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.987716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987216 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58874768-db63-47ff-b564-ff6c26f84b86-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.987716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987243 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.987716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987307 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.987716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987420 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh97q\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-kube-api-access-gh97q\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.988288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987729 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.988288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987799 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.988288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987899 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.988288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.987958 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.988288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.988192 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.988843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.988639 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58874768-db63-47ff-b564-ff6c26f84b86-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.988843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.988813 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.991306 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.991265 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.993163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.993100 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58874768-db63-47ff-b564-ff6c26f84b86-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:09.999923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:09.999818 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:10.000530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.000496 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh97q\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-kube-api-access-gh97q\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:10.079162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.079120 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:10.081205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.081176 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:10.231029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.230993 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn"] Nov 01 22:55:10.232215 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:10.232187 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7c2c33_56f7_45c5_911d_0413c08efb63.slice/crio-b396ae9fc6dc65ecd665acabe40e02e78f1eb8fff86260e40f20619614d68384 WatchSource:0}: Error finding container b396ae9fc6dc65ecd665acabe40e02e78f1eb8fff86260e40f20619614d68384: Status 404 returned error can't find the container with id b396ae9fc6dc65ecd665acabe40e02e78f1eb8fff86260e40f20619614d68384 Nov 01 22:55:10.241434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.241396 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:10.241511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.241495 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:10.241561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.241546 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:10.459452 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:10.459409 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58874768_db63_47ff_b564_ff6c26f84b86.slice/crio-2826c001011c69478b012116842e5dcfd759ef97f2cac8eb557173540c3d4aa1 WatchSource:0}: Error finding container 2826c001011c69478b012116842e5dcfd759ef97f2cac8eb557173540c3d4aa1: Status 404 returned error can't find the container with id 2826c001011c69478b012116842e5dcfd759ef97f2cac8eb557173540c3d4aa1 Nov 01 22:55:10.460217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.460191 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd"] Nov 01 22:55:10.462698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.462664 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:10.462808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.462728 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:10.462808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.462760 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:10.536381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.536341 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" event={"ID":"fc7c2c33-56f7-45c5-911d-0413c08efb63","Type":"ContainerStarted","Data":"21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6"} Nov 01 22:55:10.536532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.536405 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" event={"ID":"fc7c2c33-56f7-45c5-911d-0413c08efb63","Type":"ContainerStarted","Data":"b396ae9fc6dc65ecd665acabe40e02e78f1eb8fff86260e40f20619614d68384"} Nov 01 22:55:10.542594 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.540174 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" event={"ID":"58874768-db63-47ff-b564-ff6c26f84b86","Type":"ContainerStarted","Data":"2826c001011c69478b012116842e5dcfd759ef97f2cac8eb557173540c3d4aa1"} Nov 01 22:55:10.543317 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.543269 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" event={"ID":"1b7cafbc-700c-43eb-871f-1427195d090d","Type":"ContainerDied","Data":"8453097ecad92ca97473b14d2c74f7946bfba705a9a67f70ae362d690921e444"} Nov 01 22:55:10.543431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.543329 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr" Nov 01 22:55:10.543431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.543345 4981 scope.go:122] "RemoveContainer" containerID="c643be0b55327451feb71ddb41d7e50f82e5db204c13b379df5f47cf6238a63e" Nov 01 22:55:10.570344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.570288 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podStartSLOduration=74.57026999 podStartE2EDuration="1m14.57026999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:10.564875724 +0000 UTC m=+1875.992753668" watchObservedRunningTime="2026-11-01 22:55:10.57026999 +0000 UTC m=+1875.998147921" Nov 01 22:55:10.593923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.593881 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr"] Nov 01 22:55:10.600497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.600465 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vmhfr"] Nov 01 22:55:10.739353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.739268 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0369ac9e-dda1-4f8a-878a-24bddecac57b" path="/var/lib/kubelet/pods/0369ac9e-dda1-4f8a-878a-24bddecac57b/volumes" Nov 01 22:55:10.740186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.740162 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7cafbc-700c-43eb-871f-1427195d090d" path="/var/lib/kubelet/pods/1b7cafbc-700c-43eb-871f-1427195d090d/volumes" Nov 01 22:55:10.740839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.740810 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cabded-6252-4cb6-95c8-f0ec9024a589" path="/var/lib/kubelet/pods/a8cabded-6252-4cb6-95c8-f0ec9024a589/volumes" Nov 01 22:55:10.899602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.897500 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:10.918700 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:10.918664 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:11.437123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.437093 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 22:55:11.455025 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.454994 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf"] Nov 01 22:55:11.461830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.461806 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.466669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.466639 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:55:11.468195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.468162 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:55:11.528254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.528221 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.528701 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.528680 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.528886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.528872 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn297\" (UniqueName: \"kubernetes.io/projected/d4fd7304-5f6a-4839-a02e-999f92f44e43-kube-api-access-sn297\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.536455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.536428 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf"] Nov 01 22:55:11.563290 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.563255 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" event={"ID":"58874768-db63-47ff-b564-ff6c26f84b86","Type":"ContainerStarted","Data":"b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0"} Nov 01 22:55:11.623599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.622126 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podStartSLOduration=75.622101048 podStartE2EDuration="1m15.622101048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:11.593233362 +0000 UTC m=+1877.021111293" watchObservedRunningTime="2026-11-01 22:55:11.622101048 +0000 UTC m=+1877.049978981" Nov 01 22:55:11.631172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.631141 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.631479 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.631459 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn297\" (UniqueName: \"kubernetes.io/projected/d4fd7304-5f6a-4839-a02e-999f92f44e43-kube-api-access-sn297\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.631894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.631869 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.632440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.632408 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.641201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.641106 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn297\" (UniqueName: \"kubernetes.io/projected/d4fd7304-5f6a-4839-a02e-999f92f44e43-kube-api-access-sn297\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.644128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.644105 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2aa42aa-9qlpf\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.671393 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.671362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:11.830221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.830134 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:11.918292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.918248 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 22:55:11.918502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:11.918322 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 22:55:12.125882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.125755 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:12.128092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.127704 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 22:55:12.128092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.127763 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 22:55:12.158355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.158323 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf"] Nov 01 22:55:12.158804 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:12.158775 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fd7304_5f6a_4839_a02e_999f92f44e43.slice/crio-135310a0391fb1c48ff4021bd24289ab9777ea53ad5e6434763979ae5ee6b1e0 WatchSource:0}: Error finding container 135310a0391fb1c48ff4021bd24289ab9777ea53ad5e6434763979ae5ee6b1e0: Status 404 returned error can't find the container with id 135310a0391fb1c48ff4021bd24289ab9777ea53ad5e6434763979ae5ee6b1e0 Nov 01 22:55:12.354802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.354763 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:12.635342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.634641 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" event={"ID":"d4fd7304-5f6a-4839-a02e-999f92f44e43","Type":"ContainerStarted","Data":"135310a0391fb1c48ff4021bd24289ab9777ea53ad5e6434763979ae5ee6b1e0"} Nov 01 22:55:12.824601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.822635 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7"] Nov 01 22:55:12.846691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.837442 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7"] Nov 01 22:55:12.846691 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.837656 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:12.902601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.902352 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:12.902832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.902747 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnw5g\" (UniqueName: \"kubernetes.io/projected/3695f11f-70be-4db2-ba9d-42ae5c5f5734-kube-api-access-nnw5g\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:12.902832 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.902808 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:12.929958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.929917 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 22:55:12.930238 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:12.930211 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 22:55:13.027596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.027272 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:13.027821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.027625 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnw5g\" (UniqueName: \"kubernetes.io/projected/3695f11f-70be-4db2-ba9d-42ae5c5f5734-kube-api-access-nnw5g\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:13.027821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.027693 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:13.029034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.029006 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:13.032608 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.032547 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:13.038495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.038471 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnw5g\" (UniqueName: \"kubernetes.io/projected/3695f11f-70be-4db2-ba9d-42ae5c5f5734-kube-api-access-nnw5g\") pod \"maas-api-revoke-sub-85fae1b3-gzrm7\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:13.103390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.099464 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 22:55:13.103390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.099524 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 22:55:13.126600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.125694 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:13.168623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.159196 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:13.597913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.594718 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" podUID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" containerName="payload-processing" containerID="cri-o://c0361a14518e2227e9c8ff497a12bcedd1582a9677d4197c96e506963fbd3ca8" gracePeriod=25 Nov 01 22:55:13.597913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.594930 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" podUID="ca59b420-9ae9-4196-9cb4-c01f68908a50" containerName="payload-pre-processing" containerID="cri-o://a950c8fb96b08e416668008ecac7dc565708bf0ca5957bafddf0e8af91533e4d" gracePeriod=25 Nov 01 22:55:13.649867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.649824 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" event={"ID":"d4fd7304-5f6a-4839-a02e-999f92f44e43","Type":"ContainerStarted","Data":"002b42c35ab96d4ea3eaacc8b787af0c3f3c65acc50b4a01d33e83f484250703"} Nov 01 22:55:13.655955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.655929 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7"] Nov 01 22:55:13.657119 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:13.657097 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3695f11f_70be_4db2_ba9d_42ae5c5f5734.slice/crio-a0a79d230682bfcb62101ab4f3bada77da39da86aae61386cb8bc4acb691d8df WatchSource:0}: Error finding container a0a79d230682bfcb62101ab4f3bada77da39da86aae61386cb8bc4acb691d8df: Status 404 returned error can't find the container with id a0a79d230682bfcb62101ab4f3bada77da39da86aae61386cb8bc4acb691d8df Nov 01 22:55:13.733547 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.733485 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" podStartSLOduration=2.733462687 podStartE2EDuration="2.733462687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:13.668138736 +0000 UTC m=+1879.096016668" watchObservedRunningTime="2026-11-01 22:55:13.733462687 +0000 UTC m=+1879.161340620" Nov 01 22:55:13.751876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.751840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:13.922600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.915976 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 22:55:13.922600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:13.916037 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 22:55:14.082613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.081983 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 22:55:14.082613 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.082043 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 22:55:14.402663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.402536 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:14.501091 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.501053 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:14.634168 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:14.634134 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:55:14.661028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.660935 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" event={"ID":"3695f11f-70be-4db2-ba9d-42ae5c5f5734","Type":"ContainerStarted","Data":"fe9ba8e746e5d7f7dd12aa2c8f81040f931b651dc43ebb5382b88a1580abbe4d"} Nov 01 22:55:14.661028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.660990 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" event={"ID":"3695f11f-70be-4db2-ba9d-42ae5c5f5734","Type":"ContainerStarted","Data":"a0a79d230682bfcb62101ab4f3bada77da39da86aae61386cb8bc4acb691d8df"} Nov 01 22:55:14.663119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.663046 4981 generic.go:356] "Generic (PLEG): container finished" podID="d4fd7304-5f6a-4839-a02e-999f92f44e43" containerID="002b42c35ab96d4ea3eaacc8b787af0c3f3c65acc50b4a01d33e83f484250703" exitCode=0 Nov 01 22:55:14.663119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.663114 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" event={"ID":"d4fd7304-5f6a-4839-a02e-999f92f44e43","Type":"ContainerDied","Data":"002b42c35ab96d4ea3eaacc8b787af0c3f3c65acc50b4a01d33e83f484250703"} Nov 01 22:55:14.748469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.748035 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" podStartSLOduration=2.74802092 podStartE2EDuration="2.74802092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:14.727956976 +0000 UTC m=+1880.155834907" watchObservedRunningTime="2026-11-01 22:55:14.74802092 +0000 UTC m=+1880.175898877" Nov 01 22:55:14.927405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.923182 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 22:55:14.927405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:14.923245 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 22:55:15.098600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:15.098031 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:15.098981 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:15.098937 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 22:55:15.099109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:15.098997 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 22:55:15.691602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:15.683084 4981 generic.go:356] "Generic (PLEG): container finished" podID="3695f11f-70be-4db2-ba9d-42ae5c5f5734" containerID="fe9ba8e746e5d7f7dd12aa2c8f81040f931b651dc43ebb5382b88a1580abbe4d" exitCode=0 Nov 01 22:55:15.691602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:15.684088 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" event={"ID":"3695f11f-70be-4db2-ba9d-42ae5c5f5734","Type":"ContainerDied","Data":"fe9ba8e746e5d7f7dd12aa2c8f81040f931b651dc43ebb5382b88a1580abbe4d"} Nov 01 22:55:15.938617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:15.937526 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 22:55:15.938617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:15.937621 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 22:55:16.059565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.059532 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:16.070341 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.070318 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:16.118228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.112418 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 22:55:16.118228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.112479 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 22:55:16.361427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.361345 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-openshift-ca\") pod \"d4fd7304-5f6a-4839-a02e-999f92f44e43\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " Nov 01 22:55:16.361803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.361785 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fd7304-5f6a-4839-a02e-999f92f44e43-kube-api-access-sn297\" (UniqueName: \"kubernetes.io/projected/d4fd7304-5f6a-4839-a02e-999f92f44e43-kube-api-access-sn297\") pod \"d4fd7304-5f6a-4839-a02e-999f92f44e43\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " Nov 01 22:55:16.362000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.361982 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-xks-ca\") pod \"d4fd7304-5f6a-4839-a02e-999f92f44e43\" (UID: \"d4fd7304-5f6a-4839-a02e-999f92f44e43\") " Nov 01 22:55:16.362181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.361818 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-openshift-ca" pod "d4fd7304-5f6a-4839-a02e-999f92f44e43" (UID: "d4fd7304-5f6a-4839-a02e-999f92f44e43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:16.363412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.363386 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:16.364361 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.364329 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fd7304-5f6a-4839-a02e-999f92f44e43-kube-api-access-sn297" pod "d4fd7304-5f6a-4839-a02e-999f92f44e43" (UID: "d4fd7304-5f6a-4839-a02e-999f92f44e43"). InnerVolumeSpecName "kube-api-access-sn297". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:16.364445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.364423 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-xks-ca" pod "d4fd7304-5f6a-4839-a02e-999f92f44e43" (UID: "d4fd7304-5f6a-4839-a02e-999f92f44e43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:16.465404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.465374 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn297\" (UniqueName: \"kubernetes.io/projected/d4fd7304-5f6a-4839-a02e-999f92f44e43-kube-api-access-sn297\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:16.465404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.465403 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4fd7304-5f6a-4839-a02e-999f92f44e43-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:16.761760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.761725 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:16.765401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.765369 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" Nov 01 22:55:16.851688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.849504 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf" event={"ID":"d4fd7304-5f6a-4839-a02e-999f92f44e43","Type":"ContainerDied","Data":"135310a0391fb1c48ff4021bd24289ab9777ea53ad5e6434763979ae5ee6b1e0"} Nov 01 22:55:16.851688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.849593 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="135310a0391fb1c48ff4021bd24289ab9777ea53ad5e6434763979ae5ee6b1e0" Nov 01 22:55:16.981843 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.930166 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:17.000598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.984520 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 01 22:55:17.000598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:16.984598 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 01 22:55:17.131686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.107029 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:17.131686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.126987 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 22:55:17.131686 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.127042 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 22:55:17.281654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.281616 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:17.648596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.639343 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:17.866593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.865956 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:17.960883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.960846 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3695f11f-70be-4db2-ba9d-42ae5c5f5734-kube-api-access-nnw5g\" (UniqueName: \"kubernetes.io/projected/3695f11f-70be-4db2-ba9d-42ae5c5f5734-kube-api-access-nnw5g\") pod \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " Nov 01 22:55:17.964220 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.964191 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-openshift-ca\") pod \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " Nov 01 22:55:17.964601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.964523 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-xks-ca\") pod \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\" (UID: \"3695f11f-70be-4db2-ba9d-42ae5c5f5734\") " Nov 01 22:55:17.966987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.966216 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-openshift-ca" pod "3695f11f-70be-4db2-ba9d-42ae5c5f5734" (UID: "3695f11f-70be-4db2-ba9d-42ae5c5f5734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:17.967362 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.967336 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:17.977654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:17.977618 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3695f11f-70be-4db2-ba9d-42ae5c5f5734-kube-api-access-nnw5g" pod "3695f11f-70be-4db2-ba9d-42ae5c5f5734" (UID: "3695f11f-70be-4db2-ba9d-42ae5c5f5734"). InnerVolumeSpecName "kube-api-access-nnw5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:18.017082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.017020 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-xks-ca" pod "3695f11f-70be-4db2-ba9d-42ae5c5f5734" (UID: "3695f11f-70be-4db2-ba9d-42ae5c5f5734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:18.069809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.069772 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3695f11f-70be-4db2-ba9d-42ae5c5f5734-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:18.069809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.069811 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnw5g\" (UniqueName: \"kubernetes.io/projected/3695f11f-70be-4db2-ba9d-42ae5c5f5734-kube-api-access-nnw5g\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:18.082520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.082482 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 22:55:18.082708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.082542 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 22:55:18.209057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.209002 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:18.209646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.209609 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:18.210738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.210720 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 22:55:18.358716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.358682 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:18.850307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.850276 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" Nov 01 22:55:18.850495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.850310 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7" event={"ID":"3695f11f-70be-4db2-ba9d-42ae5c5f5734","Type":"ContainerDied","Data":"a0a79d230682bfcb62101ab4f3bada77da39da86aae61386cb8bc4acb691d8df"} Nov 01 22:55:18.850495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:18.850359 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a79d230682bfcb62101ab4f3bada77da39da86aae61386cb8bc4acb691d8df" Nov 01 22:55:19.036737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:19.036644 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" podUID="ca59b420-9ae9-4196-9cb4-c01f68908a50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:55:19.037120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:19.036644 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" podUID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:55:19.086831 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:19.086801 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:19.087266 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:19.087247 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:19.088018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:19.087997 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 22:55:19.498315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:19.498275 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:20.867350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:20.866240 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:21.737934 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:21.737900 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:22.088154 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.088069 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:22.483073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.483036 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s"] Nov 01 22:55:22.484081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.484059 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3695f11f-70be-4db2-ba9d-42ae5c5f5734" containerName="revoke-keys" Nov 01 22:55:22.484081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.484081 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3695f11f-70be-4db2-ba9d-42ae5c5f5734" containerName="revoke-keys" Nov 01 22:55:22.484250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.484108 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d4fd7304-5f6a-4839-a02e-999f92f44e43" containerName="revoke-keys" Nov 01 22:55:22.484250 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.484113 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fd7304-5f6a-4839-a02e-999f92f44e43" containerName="revoke-keys" Nov 01 22:55:22.484368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.484272 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3695f11f-70be-4db2-ba9d-42ae5c5f5734" containerName="revoke-keys" Nov 01 22:55:22.484368 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.484287 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fd7304-5f6a-4839-a02e-999f92f44e43" containerName="revoke-keys" Nov 01 22:55:22.489666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.489646 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.494444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.494365 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:55:22.495648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.495610 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:55:22.498340 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.498317 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s"] Nov 01 22:55:22.598541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.598504 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.598818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.598795 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4bd\" (UniqueName: \"kubernetes.io/projected/150f1e7b-47c7-4f19-896b-7247aadcb0e0-kube-api-access-wd4bd\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.598909 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.598895 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.701171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.701140 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4bd\" (UniqueName: \"kubernetes.io/projected/150f1e7b-47c7-4f19-896b-7247aadcb0e0-kube-api-access-wd4bd\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.701351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.701227 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.701401 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.701361 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.702142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.702112 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.703646 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.703605 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.718384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.718299 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4bd\" (UniqueName: \"kubernetes.io/projected/150f1e7b-47c7-4f19-896b-7247aadcb0e0-kube-api-access-wd4bd\") pod \"maas-api-revoke-sub-8ea4404d-xrd8s\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:22.793237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.793156 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:22.809335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:22.809310 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:23.190994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:23.190969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:23.192251 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:23.192224 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150f1e7b_47c7_4f19_896b_7247aadcb0e0.slice/crio-b53029ff721f912f65251240bef9f4a90d1379619e2e9dd05e3e3fbeccc199a5 WatchSource:0}: Error finding container b53029ff721f912f65251240bef9f4a90d1379619e2e9dd05e3e3fbeccc199a5: Status 404 returned error can't find the container with id b53029ff721f912f65251240bef9f4a90d1379619e2e9dd05e3e3fbeccc199a5 Nov 01 22:55:23.192513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:23.192492 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s"] Nov 01 22:55:23.569085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:23.569001 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:23.850741 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:23.850659 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:23.948615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:23.948547 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" event={"ID":"150f1e7b-47c7-4f19-896b-7247aadcb0e0","Type":"ContainerStarted","Data":"66e53ef525cad2e30c6b34082dc7fc062ccb9dad3c657c5d9272ce93f4a8398a"} Nov 01 22:55:23.948813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:23.948685 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" event={"ID":"150f1e7b-47c7-4f19-896b-7247aadcb0e0","Type":"ContainerStarted","Data":"b53029ff721f912f65251240bef9f4a90d1379619e2e9dd05e3e3fbeccc199a5"} Nov 01 22:55:24.007600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.000977 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" podStartSLOduration=2.000955298 podStartE2EDuration="2.000955298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:23.996138287 +0000 UTC m=+1889.424016221" watchObservedRunningTime="2026-11-01 22:55:24.000955298 +0000 UTC m=+1889.428833231" Nov 01 22:55:24.387506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.387469 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx"] Nov 01 22:55:24.393303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.393270 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.401640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.401613 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx"] Nov 01 22:55:24.464896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.464858 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.465259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.465237 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfps\" (UniqueName: \"kubernetes.io/projected/afb0284b-633b-47f0-a13a-fb8aa9f38fce-kube-api-access-8wfps\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.465441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.465407 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.567435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.567325 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfps\" (UniqueName: \"kubernetes.io/projected/afb0284b-633b-47f0-a13a-fb8aa9f38fce-kube-api-access-8wfps\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.567630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.567487 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.567630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.567604 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.568409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.568384 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.570249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.570225 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.576518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.576498 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfps\" (UniqueName: \"kubernetes.io/projected/afb0284b-633b-47f0-a13a-fb8aa9f38fce-kube-api-access-8wfps\") pod \"maas-api-revoke-sub-59afc768-nkjkx\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.756597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.755298 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:24.761702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.756864 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:24.960638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.960564 4981 generic.go:356] "Generic (PLEG): container finished" podID="150f1e7b-47c7-4f19-896b-7247aadcb0e0" containerID="66e53ef525cad2e30c6b34082dc7fc062ccb9dad3c657c5d9272ce93f4a8398a" exitCode=0 Nov 01 22:55:24.960872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:24.960656 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" event={"ID":"150f1e7b-47c7-4f19-896b-7247aadcb0e0","Type":"ContainerDied","Data":"66e53ef525cad2e30c6b34082dc7fc062ccb9dad3c657c5d9272ce93f4a8398a"} Nov 01 22:55:25.065455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.065422 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:25.067744 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:25.067717 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb0284b_633b_47f0_a13a_fb8aa9f38fce.slice/crio-8a081a069a5f46eac82a251c4802467dae47702881e80a6f87f0f9658617ac5a WatchSource:0}: Error finding container 8a081a069a5f46eac82a251c4802467dae47702881e80a6f87f0f9658617ac5a: Status 404 returned error can't find the container with id 8a081a069a5f46eac82a251c4802467dae47702881e80a6f87f0f9658617ac5a Nov 01 22:55:25.068473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.068447 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx"] Nov 01 22:55:25.174451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.174390 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:25.383903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.383832 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:25.934195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.934160 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:25.971147 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.971113 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" event={"ID":"afb0284b-633b-47f0-a13a-fb8aa9f38fce","Type":"ContainerStarted","Data":"459390a1f57a294494e4ad60d45f7db1241c2e554721b68bb8ad1c29dd69df5a"} Nov 01 22:55:25.971309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.971166 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" event={"ID":"afb0284b-633b-47f0-a13a-fb8aa9f38fce","Type":"ContainerStarted","Data":"8a081a069a5f46eac82a251c4802467dae47702881e80a6f87f0f9658617ac5a"} Nov 01 22:55:25.993360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:25.993310 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" podStartSLOduration=1.993297606 podStartE2EDuration="1.993297606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:25.989553651 +0000 UTC m=+1891.417431583" watchObservedRunningTime="2026-11-01 22:55:25.993297606 +0000 UTC m=+1891.421175536" Nov 01 22:55:26.037388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.037309 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:26.191619 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.191531 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:26.302458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.302416 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-openshift-ca\") pod \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " Nov 01 22:55:26.302654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.302552 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150f1e7b-47c7-4f19-896b-7247aadcb0e0-kube-api-access-wd4bd\" (UniqueName: \"kubernetes.io/projected/150f1e7b-47c7-4f19-896b-7247aadcb0e0-kube-api-access-wd4bd\") pod \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " Nov 01 22:55:26.302849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.302828 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-xks-ca\") pod \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\" (UID: \"150f1e7b-47c7-4f19-896b-7247aadcb0e0\") " Nov 01 22:55:26.304071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.304041 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-openshift-ca" pod "150f1e7b-47c7-4f19-896b-7247aadcb0e0" (UID: "150f1e7b-47c7-4f19-896b-7247aadcb0e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:26.306959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.306886 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-xks-ca" pod "150f1e7b-47c7-4f19-896b-7247aadcb0e0" (UID: "150f1e7b-47c7-4f19-896b-7247aadcb0e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:26.308259 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.308233 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150f1e7b-47c7-4f19-896b-7247aadcb0e0-kube-api-access-wd4bd" pod "150f1e7b-47c7-4f19-896b-7247aadcb0e0" (UID: "150f1e7b-47c7-4f19-896b-7247aadcb0e0"). InnerVolumeSpecName "kube-api-access-wd4bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:26.378237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.378201 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2"] Nov 01 22:55:26.378441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.378418 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" containerID="cri-o://2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7" gracePeriod=30 Nov 01 22:55:26.381475 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.381451 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:26.417004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.416972 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:26.417004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.417004 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/150f1e7b-47c7-4f19-896b-7247aadcb0e0-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:26.417198 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.417022 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4bd\" (UniqueName: \"kubernetes.io/projected/150f1e7b-47c7-4f19-896b-7247aadcb0e0-kube-api-access-wd4bd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:26.510660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.510326 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:26.991092 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.991055 4981 generic.go:356] "Generic (PLEG): container finished" podID="afb0284b-633b-47f0-a13a-fb8aa9f38fce" containerID="459390a1f57a294494e4ad60d45f7db1241c2e554721b68bb8ad1c29dd69df5a" exitCode=0 Nov 01 22:55:26.991548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.991134 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" event={"ID":"afb0284b-633b-47f0-a13a-fb8aa9f38fce","Type":"ContainerDied","Data":"459390a1f57a294494e4ad60d45f7db1241c2e554721b68bb8ad1c29dd69df5a"} Nov 01 22:55:26.992622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.992602 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" Nov 01 22:55:26.992740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.992601 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s" event={"ID":"150f1e7b-47c7-4f19-896b-7247aadcb0e0","Type":"ContainerDied","Data":"b53029ff721f912f65251240bef9f4a90d1379619e2e9dd05e3e3fbeccc199a5"} Nov 01 22:55:26.992740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:26.992717 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b53029ff721f912f65251240bef9f4a90d1379619e2e9dd05e3e3fbeccc199a5" Nov 01 22:55:27.090186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:27.090095 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:27.280514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:27.280474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:27.648595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:27.645156 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:28.066271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.066237 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr"] Nov 01 22:55:28.066664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.066475 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerName="discovery" containerID="cri-o://466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6" gracePeriod=30 Nov 01 22:55:28.116397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.116278 4981 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-h9fzr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.221:8080/ready\": dial tcp 10.134.0.221:8080: connect: connection refused" start-of-body= Nov 01 22:55:28.116397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.116331 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.221:8080/ready\": dial tcp 10.134.0.221:8080: connect: connection refused" Nov 01 22:55:28.347328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.342088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:28.382507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.382476 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:28.450044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.450013 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:55:28.461117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.461082 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb0284b-633b-47f0-a13a-fb8aa9f38fce-kube-api-access-8wfps\" (UniqueName: \"kubernetes.io/projected/afb0284b-633b-47f0-a13a-fb8aa9f38fce-kube-api-access-8wfps\") pod \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " Nov 01 22:55:28.461278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.461211 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-openshift-ca\") pod \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " Nov 01 22:55:28.461346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.461321 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-xks-ca\") pod \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\" (UID: \"afb0284b-633b-47f0-a13a-fb8aa9f38fce\") " Nov 01 22:55:28.461557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.461518 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-openshift-ca" pod "afb0284b-633b-47f0-a13a-fb8aa9f38fce" (UID: "afb0284b-633b-47f0-a13a-fb8aa9f38fce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:28.463537 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.463508 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb0284b-633b-47f0-a13a-fb8aa9f38fce-kube-api-access-8wfps" pod "afb0284b-633b-47f0-a13a-fb8aa9f38fce" (UID: "afb0284b-633b-47f0-a13a-fb8aa9f38fce"). InnerVolumeSpecName "kube-api-access-8wfps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:28.463974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.463944 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-xks-ca" pod "afb0284b-633b-47f0-a13a-fb8aa9f38fce" (UID: "afb0284b-633b-47f0-a13a-fb8aa9f38fce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:28.464190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.464176 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wfps\" (UniqueName: \"kubernetes.io/projected/afb0284b-633b-47f0-a13a-fb8aa9f38fce-kube-api-access-8wfps\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.464235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.464192 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.464235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.464202 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb0284b-633b-47f0-a13a-fb8aa9f38fce-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.503904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.503875 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4"] Nov 01 22:55:28.505437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505383 4981 cpu_manager.go:414] "containerMap: removing container" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerName="discovery" Nov 01 22:55:28.505437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505412 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerName="discovery" Nov 01 22:55:28.505437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505435 4981 cpu_manager.go:414] "containerMap: removing container" podUID="afb0284b-633b-47f0-a13a-fb8aa9f38fce" containerName="revoke-keys" Nov 01 22:55:28.505437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505442 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb0284b-633b-47f0-a13a-fb8aa9f38fce" containerName="revoke-keys" Nov 01 22:55:28.507836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505505 4981 cpu_manager.go:414] "containerMap: removing container" podUID="150f1e7b-47c7-4f19-896b-7247aadcb0e0" containerName="revoke-keys" Nov 01 22:55:28.507836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505512 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150f1e7b-47c7-4f19-896b-7247aadcb0e0" containerName="revoke-keys" Nov 01 22:55:28.507836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505798 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerName="discovery" Nov 01 22:55:28.507836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505815 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb0284b-633b-47f0-a13a-fb8aa9f38fce" containerName="revoke-keys" Nov 01 22:55:28.507836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.505833 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="150f1e7b-47c7-4f19-896b-7247aadcb0e0" containerName="revoke-keys" Nov 01 22:55:28.512328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.512303 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.519673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.517000 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb"] Nov 01 22:55:28.523414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.523390 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-49ab43-openshift-default-dockercfg-tnrmv\"" Nov 01 22:55:28.525665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.525642 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n"] Nov 01 22:55:28.525889 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.525867 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.533324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.533304 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4"] Nov 01 22:55:28.534961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.533504 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.557706 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.557677 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7"] Nov 01 22:55:28.561920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.561864 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.565268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.565251 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4"] Nov 01 22:55:28.565446 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.565436 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.571305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.571278 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-kubeconfig\") pod \"891d2984-c0a2-43c0-9d7a-62e7212486a3\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " Nov 01 22:55:28.571403 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.571369 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-cacerts\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-cacerts\") pod \"891d2984-c0a2-43c0-9d7a-62e7212486a3\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " Nov 01 22:55:28.571614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.571594 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-kube-api-access-b267g\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-kube-api-access-b267g\") pod \"891d2984-c0a2-43c0-9d7a-62e7212486a3\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " Nov 01 22:55:28.571727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.571697 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-ca-configmap\") pod \"891d2984-c0a2-43c0-9d7a-62e7212486a3\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " Nov 01 22:55:28.571829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.571802 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-dns-cert\") pod \"891d2984-c0a2-43c0-9d7a-62e7212486a3\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " Nov 01 22:55:28.571919 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.571903 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-token\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-token\") pod \"891d2984-c0a2-43c0-9d7a-62e7212486a3\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " Nov 01 22:55:28.572113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.572095 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891d2984-c0a2-43c0-9d7a-62e7212486a3-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/891d2984-c0a2-43c0-9d7a-62e7212486a3-local-certs\") pod \"891d2984-c0a2-43c0-9d7a-62e7212486a3\" (UID: \"891d2984-c0a2-43c0-9d7a-62e7212486a3\") " Nov 01 22:55:28.573366 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.573126 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-ca-configmap" pod "891d2984-c0a2-43c0-9d7a-62e7212486a3" (UID: "891d2984-c0a2-43c0-9d7a-62e7212486a3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:28.577013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.576981 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n"] Nov 01 22:55:28.582567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.582504 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb"] Nov 01 22:55:28.583542 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.583509 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-kubeconfig" pod "891d2984-c0a2-43c0-9d7a-62e7212486a3" (UID: "891d2984-c0a2-43c0-9d7a-62e7212486a3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:28.585652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.585546 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-token" pod "891d2984-c0a2-43c0-9d7a-62e7212486a3" (UID: "891d2984-c0a2-43c0-9d7a-62e7212486a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:28.585756 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.585705 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891d2984-c0a2-43c0-9d7a-62e7212486a3-local-certs" pod "891d2984-c0a2-43c0-9d7a-62e7212486a3" (UID: "891d2984-c0a2-43c0-9d7a-62e7212486a3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:28.586161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.586128 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-kube-api-access-b267g" pod "891d2984-c0a2-43c0-9d7a-62e7212486a3" (UID: "891d2984-c0a2-43c0-9d7a-62e7212486a3"). InnerVolumeSpecName "kube-api-access-b267g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:28.587923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.587698 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-dns-cert" pod "891d2984-c0a2-43c0-9d7a-62e7212486a3" (UID: "891d2984-c0a2-43c0-9d7a-62e7212486a3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:28.604206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.604171 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7"] Nov 01 22:55:28.615435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.615405 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4"] Nov 01 22:55:28.618213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.618015 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-cacerts" pod "891d2984-c0a2-43c0-9d7a-62e7212486a3" (UID: "891d2984-c0a2-43c0-9d7a-62e7212486a3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:28.675402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675365 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84x8\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-kube-api-access-f84x8\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.675402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675410 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.675684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675458 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.675684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675484 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.675684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675566 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.675684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675668 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqvc2\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-kube-api-access-tqvc2\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.675847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675708 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.675847 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675803 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-socket\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.675954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675844 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.675954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675883 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.676070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.675975 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.676070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676006 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a51428-3f39-46e6-8057-6518fce34f4d-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.676070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676035 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.676203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676081 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-certs\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.676203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676134 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.676203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676175 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.676335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676212 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.676335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676249 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.676335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676269 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.676335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676287 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995ecca3-6286-489f-ba37-a6170687b0c3-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.676476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676397 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995ecca3-6286-489f-ba37-a6170687b0c3-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.676476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676460 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51812fed-7df5-42f0-9e69-70130fe330bd-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.676563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676506 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-data\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.676563 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676554 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.676685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676599 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rsb\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-kube-api-access-k9rsb\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.676685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676671 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgst\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-kube-api-access-hcgst\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.676768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676713 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a708cdd3-2466-4846-876a-99b7f2c35bfc-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.676768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676743 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.676841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676816 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.676885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676854 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-envoy\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.676920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.676991 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.676975 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6n6z\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-kube-api-access-q6n6z\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.677028 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677009 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51812fed-7df5-42f0-9e69-70130fe330bd-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.677121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677106 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.677162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677153 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-podinfo\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.677200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677176 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.677272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677257 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.677309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677285 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-token\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.677346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677309 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.677392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677376 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-credential-socket\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.677437 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677424 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.677532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677517 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.677634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677552 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.677634 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677590 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.677746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677645 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istiod-ca-cert\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.677824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677810 4981 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/891d2984-c0a2-43c0-9d7a-62e7212486a3-local-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.677897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677832 4981 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-kubeconfig\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.677897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677848 4981 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-cacerts\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.677897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677875 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b267g\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-kube-api-access-b267g\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.677897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677892 4981 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-ca-configmap\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.678047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677907 4981 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-csr-dns-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.678047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.677924 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891d2984-c0a2-43c0-9d7a-62e7212486a3-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:28.779151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779111 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51812fed-7df5-42f0-9e69-70130fe330bd-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.779151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779151 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-data\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.779402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779172 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.779402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779219 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rsb\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-kube-api-access-k9rsb\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.779402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779269 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgst\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-kube-api-access-hcgst\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.779402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779297 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a708cdd3-2466-4846-876a-99b7f2c35bfc-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.779402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779320 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.779713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779641 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-data\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.779713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779691 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.779713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779708 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.779876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779719 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-data\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.779876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779772 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-envoy\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.779876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779844 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.780047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779887 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6n6z\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-kube-api-access-q6n6z\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.780047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779913 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51812fed-7df5-42f0-9e69-70130fe330bd-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.780047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.779984 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.780203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780050 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-podinfo\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.780203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780077 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.780203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780144 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.780203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780160 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a708cdd3-2466-4846-876a-99b7f2c35bfc-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.780203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780177 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-token\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.780532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780225 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.780532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780273 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-credential-socket\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.780532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780336 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.780532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780415 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.780532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780422 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780571 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780624 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780653 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istiod-ca-cert\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780687 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84x8\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-kube-api-access-f84x8\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780710 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780745 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51812fed-7df5-42f0-9e69-70130fe330bd-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780749 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.780937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780746 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780950 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780984 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780986 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-credential-socket\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.780987 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781061 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvc2\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-kube-api-access-tqvc2\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781106 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781149 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-socket\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781199 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781233 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781248 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781318 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.781373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781348 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a51428-3f39-46e6-8057-6518fce34f4d-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781385 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781421 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-certs\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781456 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781460 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781491 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781520 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781564 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781646 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781681 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995ecca3-6286-489f-ba37-a6170687b0c3-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781773 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995ecca3-6286-489f-ba37-a6170687b0c3-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.781910 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istiod-ca-cert\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.782031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.782007 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.782722 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.782083 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.783539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.783250 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-socket\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.783539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.783288 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.783539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.783327 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-credential-socket\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.783539 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.783405 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a51428-3f39-46e6-8057-6518fce34f4d-istiod-ca-cert\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.784836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.783566 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-certs\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.784836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.783788 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-envoy\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.784836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.784103 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51812fed-7df5-42f0-9e69-70130fe330bd-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.784836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.784644 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-podinfo\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.784836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.784779 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-certs\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.785129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.785040 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.785458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.785435 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-credential-socket\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.785992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.785720 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.785992 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.785953 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-certs\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.786278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.786250 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995ecca3-6286-489f-ba37-a6170687b0c3-istiod-ca-cert\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.786383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.786326 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-data\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.786651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.786629 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.786750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.786662 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-envoy\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.786750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.786665 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-envoy\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.786927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.786906 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.787073 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.787054 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-podinfo\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.788113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.788093 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995ecca3-6286-489f-ba37-a6170687b0c3-istio-podinfo\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.793781 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.793756 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgst\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-kube-api-access-hcgst\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.795359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.795314 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.795856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.795780 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6n6z\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-kube-api-access-q6n6z\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.796852 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.796827 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.796967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.796894 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rsb\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-kube-api-access-k9rsb\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.797837 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.797813 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-token\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.798020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.798004 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-istio-token\") pod \"e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:28.798384 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.798362 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84x8\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-kube-api-access-f84x8\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.798777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.798761 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqvc2\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-kube-api-access-tqvc2\") pod \"e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.802292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.802267 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-token\") pod \"e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.803676 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.803649 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:28.849545 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.843201 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:28.872156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.872124 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:28.917488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.917203 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:28.920499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.920461 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:28.943808 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:28.932959 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:29.037754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.037387 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" podUID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:55:29.042280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.042195 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" podUID="ca59b420-9ae9-4196-9cb4-c01f68908a50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:55:29.052473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.051069 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" event={"ID":"afb0284b-633b-47f0-a13a-fb8aa9f38fce","Type":"ContainerDied","Data":"8a081a069a5f46eac82a251c4802467dae47702881e80a6f87f0f9658617ac5a"} Nov 01 22:55:29.052473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.051128 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a081a069a5f46eac82a251c4802467dae47702881e80a6f87f0f9658617ac5a" Nov 01 22:55:29.052473 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.051221 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx" Nov 01 22:55:29.058074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.058045 4981 generic.go:356] "Generic (PLEG): container finished" podID="891d2984-c0a2-43c0-9d7a-62e7212486a3" containerID="466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6" exitCode=0 Nov 01 22:55:29.058216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.058093 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" event={"ID":"891d2984-c0a2-43c0-9d7a-62e7212486a3","Type":"ContainerDied","Data":"466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6"} Nov 01 22:55:29.058216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.058135 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" event={"ID":"891d2984-c0a2-43c0-9d7a-62e7212486a3","Type":"ContainerDied","Data":"4b040d6f17fd404933a372e4a1c2650d4663d0ce5bf14f2ad1978b39daa79149"} Nov 01 22:55:29.058216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.058160 4981 scope.go:122] "RemoveContainer" containerID="466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6" Nov 01 22:55:29.058216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.058176 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr" Nov 01 22:55:29.107903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.107605 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr"] Nov 01 22:55:29.120918 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.120890 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h9fzr"] Nov 01 22:55:29.161939 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.161914 4981 scope.go:122] "RemoveContainer" containerID="466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6" Nov 01 22:55:29.162302 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:29.162278 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6\": container with ID starting with 466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6 not found: ID does not exist" containerID="466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6" Nov 01 22:55:29.162392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.162313 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6"} err="failed to get container status \"466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6\": rpc error: code = NotFound desc = could not find container \"466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6\": container with ID starting with 466834ecf3d5dc276fc7dc0984b570ea61bc34ffd1228debc2107a5e929c0dd6 not found: ID does not exist" Nov 01 22:55:29.214396 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:29.214352 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b1f5e_3ca2_4a32_bd38_724d06a1de6b.slice/crio-3e31280f38f70279a12ec42b3b042bfcd04f68a89529f284e414f2bdb62a7f46 WatchSource:0}: Error finding container 3e31280f38f70279a12ec42b3b042bfcd04f68a89529f284e414f2bdb62a7f46: Status 404 returned error can't find the container with id 3e31280f38f70279a12ec42b3b042bfcd04f68a89529f284e414f2bdb62a7f46 Nov 01 22:55:29.215444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.215420 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4"] Nov 01 22:55:29.222146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.222103 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.222302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.222202 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.222302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.222246 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.245217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.245187 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb"] Nov 01 22:55:29.248159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.248127 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.248258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.248204 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.248258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.248249 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.332633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.332562 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n"] Nov 01 22:55:29.333316 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:29.333082 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a51428_3f39_46e6_8057_6518fce34f4d.slice/crio-cb7057feebd804560227059aa8ce7f510e2fd6531f2f95650a89306ae3c60954 WatchSource:0}: Error finding container cb7057feebd804560227059aa8ce7f510e2fd6531f2f95650a89306ae3c60954: Status 404 returned error can't find the container with id cb7057feebd804560227059aa8ce7f510e2fd6531f2f95650a89306ae3c60954 Nov 01 22:55:29.337079 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:29.337056 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995ecca3_6286_489f_ba37_a6170687b0c3.slice/crio-23c2263577960156eff1a13d4b080ed24e1b9289c58ec3b37fe7c4b98084ebe5 WatchSource:0}: Error finding container 23c2263577960156eff1a13d4b080ed24e1b9289c58ec3b37fe7c4b98084ebe5: Status 404 returned error can't find the container with id 23c2263577960156eff1a13d4b080ed24e1b9289c58ec3b37fe7c4b98084ebe5 Nov 01 22:55:29.337240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.337209 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.337315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.337282 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.337370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.337312 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.341386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.340278 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7"] Nov 01 22:55:29.352245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.350410 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4"] Nov 01 22:55:29.352245 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:29.350694 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51812fed_7df5_42f0_9e69_70130fe330bd.slice/crio-b733aa8808bef1a72a92534ed40519566127635084dd28babd78612da63f514c WatchSource:0}: Error finding container b733aa8808bef1a72a92534ed40519566127635084dd28babd78612da63f514c: Status 404 returned error can't find the container with id b733aa8808bef1a72a92534ed40519566127635084dd28babd78612da63f514c Nov 01 22:55:29.352245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.350966 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.352245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.351031 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.352245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.351082 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.360669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.360594 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.360779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.360676 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:29.360779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:29.360725 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:55:30.033910 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.033875 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:30.173294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.173251 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" event={"ID":"995ecca3-6286-489f-ba37-a6170687b0c3","Type":"ContainerStarted","Data":"3c526cefccca3201376b767ead359b6ae9d729563cf4e3c0ae20772c8e45668f"} Nov 01 22:55:30.174841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.173320 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" event={"ID":"995ecca3-6286-489f-ba37-a6170687b0c3","Type":"ContainerStarted","Data":"23c2263577960156eff1a13d4b080ed24e1b9289c58ec3b37fe7c4b98084ebe5"} Nov 01 22:55:30.178954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.178916 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" event={"ID":"d1a51428-3f39-46e6-8057-6518fce34f4d","Type":"ContainerStarted","Data":"053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871"} Nov 01 22:55:30.179102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.178993 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" event={"ID":"d1a51428-3f39-46e6-8057-6518fce34f4d","Type":"ContainerStarted","Data":"cb7057feebd804560227059aa8ce7f510e2fd6531f2f95650a89306ae3c60954"} Nov 01 22:55:30.183740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.183689 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" event={"ID":"a708cdd3-2466-4846-876a-99b7f2c35bfc","Type":"ContainerStarted","Data":"d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957"} Nov 01 22:55:30.183906 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.183753 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" event={"ID":"a708cdd3-2466-4846-876a-99b7f2c35bfc","Type":"ContainerStarted","Data":"d81023c7b86eb40bf1bb785f28d1db8a70fc967e7356c14c44ec0982b0aeaeff"} Nov 01 22:55:30.188592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.186223 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" event={"ID":"51812fed-7df5-42f0-9e69-70130fe330bd","Type":"ContainerStarted","Data":"96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025"} Nov 01 22:55:30.188592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.186276 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" event={"ID":"51812fed-7df5-42f0-9e69-70130fe330bd","Type":"ContainerStarted","Data":"b733aa8808bef1a72a92534ed40519566127635084dd28babd78612da63f514c"} Nov 01 22:55:30.189751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.189646 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" event={"ID":"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b","Type":"ContainerStarted","Data":"c4db3e1058867c2a345a8173ebbe5ab99760039ae8f872ec0b581e113bb0073f"} Nov 01 22:55:30.189946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.189927 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" event={"ID":"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b","Type":"ContainerStarted","Data":"3e31280f38f70279a12ec42b3b042bfcd04f68a89529f284e414f2bdb62a7f46"} Nov 01 22:55:30.245014 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.244946 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podStartSLOduration=277.244925067 podStartE2EDuration="4m37.244925067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:30.22186116 +0000 UTC m=+1895.649739086" watchObservedRunningTime="2026-11-01 22:55:30.244925067 +0000 UTC m=+1895.672802999" Nov 01 22:55:30.297652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.274005 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podStartSLOduration=79.273976817 podStartE2EDuration="1m19.273976817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:30.255366333 +0000 UTC m=+1895.683244264" watchObservedRunningTime="2026-11-01 22:55:30.273976817 +0000 UTC m=+1895.701854750" Nov 01 22:55:30.357814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.357746 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podStartSLOduration=3.357723439 podStartE2EDuration="3.357723439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:30.329630469 +0000 UTC m=+1895.757508406" watchObservedRunningTime="2026-11-01 22:55:30.357723439 +0000 UTC m=+1895.785601371" Nov 01 22:55:30.390201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.390115 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podStartSLOduration=79.390088343 podStartE2EDuration="1m19.390088343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:30.36232179 +0000 UTC m=+1895.790199748" watchObservedRunningTime="2026-11-01 22:55:30.390088343 +0000 UTC m=+1895.817966275" Nov 01 22:55:30.522782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.522716 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podStartSLOduration=79.522695557 podStartE2EDuration="1m19.522695557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:30.461065995 +0000 UTC m=+1895.888943927" watchObservedRunningTime="2026-11-01 22:55:30.522695557 +0000 UTC m=+1895.950573491" Nov 01 22:55:30.887767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.887720 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:30.888060 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.888012 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:30.974373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.974327 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:30.974688 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.974659 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:30.989805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.989733 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:30.990142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:30.990107 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:31.006467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.006432 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:31.028612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.026502 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:31.028612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.026563 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:31.047672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.047232 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891d2984-c0a2-43c0-9d7a-62e7212486a3" path="/var/lib/kubelet/pods/891d2984-c0a2-43c0-9d7a-62e7212486a3/volumes" Nov 01 22:55:31.049625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.047852 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:31.049625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.047910 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:31.049625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.047992 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:31.049625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.048016 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:31.049625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.048069 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:31.049625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.048114 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:31.274048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.273177 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:31.302445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.301240 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:55:31.523602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.516645 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 01 22:55:31.555117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.529707 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:55:31.565624 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.555726 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:31.603645 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.583023 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:31.728603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.703131 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:55:31.917926 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.917814 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:55:31.948983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.948927 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:32.351292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.351240 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:32.380603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.025015 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:32.380603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.035876 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:32.380603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.087361 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:55:32.380603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.140931 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:32.380603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.141679 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:55:32.380603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.970654 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:55:32.418979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.952256 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:32.468959 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.468828 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:32.469209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:31.971909 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:32.469442 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.469417 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:32.469736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.413285 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:32.469894 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.469870 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:32.520049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.520006 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:32.591967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.591930 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:32.636725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.636631 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:32.638471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.638441 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:55:32.670167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.540072 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:32.671444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.670831 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:55:32.736601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.726006 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:55:32.736601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.730282 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:55:32.739677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.739647 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:32.867598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.867536 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:55:32.902526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.900232 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:32.968392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.968348 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:32.969070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.937048 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:32.969791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.936169 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:32.969791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.969682 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:32.969791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.967941 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:32.969791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.969746 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:33.008602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.002796 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:33.008602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.002863 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:33.027617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.012974 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:33.031424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:32.636560 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:55:33.137388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.137347 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.337597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.314056 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.389228 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.389182 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.390167 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.390127 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.393824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.391555 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:55:33.402644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.391610 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:55:33.408994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.391630 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:55:33.492767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.459745 4981 patch_prober.go:29] interesting pod/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Nov 01 22:55:33.492767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.459801 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Nov 01 22:55:33.531078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.524781 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.550424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.550290 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.567540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.567497 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.631166 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.631130 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:55:33.642606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.638108 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:55:33.762795 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.762764 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:55:33.798863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.798831 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:55:33.850738 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.850706 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:55:33.851018 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.851003 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:55:33.851168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.851156 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:55:33.851857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.728333 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:55:33.880383 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.654539 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.880862 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.723870 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.900348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.825360 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 01 22:55:33.900348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.874614 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:33.900348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.897212 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:33.900348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.897984 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:33.900348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.898344 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:55:33.919152 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.919119 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:33.920050 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.920018 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:33.927717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:33.927685 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:34.201229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.201186 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:34.203648 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.062100 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.207122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.161349 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:55:34.362616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.062041 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.374708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.059979 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:34.374708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.365108 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:34.374708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.367646 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:55:34.374708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.367752 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:55:34.391601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.390887 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.417902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.402670 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:55:34.417902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.403082 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:55:34.417902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.403346 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:55:34.513998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.425503 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.513998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.476266 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.619690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.618163 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.731348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.724281 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:55:34.762140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.762099 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:55:34.899683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.896831 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.899683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.897801 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:55:34.899683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.898088 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.899683 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.899374 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:34.942561 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.823675 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:34.943544 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.943521 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:55:34.958493 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.958458 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:34.958737 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.958708 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:34.980067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.968594 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:55:34.980067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.968918 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 01 22:55:34.980067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.968976 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Nov 01 22:55:34.989116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.989087 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:34.989657 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.989624 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:35.022598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:34.986671 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:35.258611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.232642 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:35.258611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.232747 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:35.258611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.232957 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:35.258611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.233002 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:35.258611 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.172548 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:35.392995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.331221 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:55:35.392995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.352779 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:35.392995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.353327 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:35.514623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.514493 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:35.516472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.516439 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:35.517947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.517922 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:35.552604 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.552548 4981 generic.go:356] "Generic (PLEG): container finished" podID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerID="2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7" exitCode=0 Nov 01 22:55:35.568699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.565662 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.711878 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.423120 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.442884 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.454751 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.457929 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.713897 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.795838 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.952415 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:35.961015 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.497284 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.347004 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.497356 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.062029 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.212807 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" podUID="7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.499466 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.499523 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.526825 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.537774 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.573187 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.561390 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:55:36.628620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.588166 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:55:36.656751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.647245 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:55:36.656751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.648386 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.770600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.769702 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:55:36.795865 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.795815 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Nov 01 22:55:36.868665 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.862002 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 01 22:55:36.920776 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.890087 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:36.981541 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.981493 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:37.296814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.052613 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:37.297412 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.052679 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:37.306840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.306785 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:37.597651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.453857 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:37.598485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.297183 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:37.623743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.468970 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:55:37.624286 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:36.060614 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-gzxqz" podUID="fd1ade81-64b5-4dcf-a5da-55632908568b" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 01 22:55:37.624472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.234710 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" podUID="c6f43edf-54c6-41d3-8445-a8d2aa81a230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:37.717165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.692722 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:55:37.830600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.824686 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" podUID="7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:37.831300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.831261 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:37.831603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.831550 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:38.347875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.850750 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.347875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.338210 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:38.399876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.373869 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-r44kc" podUID="b224070e-06e8-4c40-afdf-2aa6765aae2f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.20:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:38.399876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.909072 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:55:38.399876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.909157 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:55:38.406603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.909192 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:55:38.406603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.909265 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:55:38.410864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.121171 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:55:38.437451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.121212 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:55:38.437888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.121284 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:55:38.447755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.309142 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:38.472217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.472150 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:55:38.473888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.473836 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:55:38.562601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.908904 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:55:38.562601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:37.876544 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:55:38.562601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.553291 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.562601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.553337 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:38.563345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.563307 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:55:38.580024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.569314 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:38.631685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.630770 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" podUID="c6f43edf-54c6-41d3-8445-a8d2aa81a230" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:38.631685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.630917 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:55:38.631685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.631149 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.631685 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.631186 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:38.693947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.693701 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:38.811938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.811823 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:55:38.887956 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.887840 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.941830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.941773 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Nov 01 22:55:38.942116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.903293 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.152:50052\" within 1s: context deadline exceeded" Nov 01 22:55:38.942278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.907453 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.214:50052\" within 1s: context deadline exceeded" Nov 01 22:55:38.969770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.908213 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn" podUID="7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.911664 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.210:50052\" within 1s: context deadline exceeded" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.901716 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.211:50052\" within 1s: context deadline exceeded" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:38.901922 4981 logging.go:55] [core] [Channel #7262 SubChannel #7267]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.211:50052", ServerName: "10.134.0.211:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.211:50052: operation was canceled" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.911734 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.971919 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:38.923511 4981 logging.go:55] [core] [Channel #7261 SubChannel #7271]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.152:50052", ServerName: "10.134.0.152:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.152:50052: operation was canceled" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.924068 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.972204 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:38.931947 4981 logging.go:55] [core] [Channel #7264 SubChannel #7272]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.214:50052", ServerName: "10.134.0.214:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:41796->10.134.0.214:50052: use of closed network connection" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.936453 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.972276 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:38.972681 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.985s" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.972712 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" event={"ID":"b1489db7-ad64-4845-8fdc-2474a55aaf0e","Type":"ContainerDied","Data":"2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7"} Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.972805 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.972830 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m"] Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.973567 4981 patch_prober.go:29] interesting pod/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974171 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974260 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974281 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974329 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974344 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974398 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974406 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" Nov 01 22:55:38.987478 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.974696 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" Nov 01 22:55:38.989043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.988117 4981 scope.go:122] "RemoveContainer" containerID="2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7" Nov 01 22:55:38.989043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:38.988252 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.040598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.026332 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:55:39.040598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.026807 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:55:39.050240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.050204 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:55:39.055433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.055409 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m"] Nov 01 22:55:39.083351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.082642 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:55:39.112763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.112725 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-socket\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.112957 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.112816 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-certs\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.113079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.113002 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-kube-api-access-b8kjx\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-kube-api-access-b8kjx\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.113246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.113224 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.113408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.113381 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-podinfo\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.113630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.113612 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:55:39.113861 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.113828 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.123801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.123670 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-socket" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.127294 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.127267 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:55:39.127416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.127390 4981 patch_prober.go:29] interesting pod/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:55:39.127491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.127430 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:39.127718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.127695 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-certs" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.174651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.154128 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:39.179026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.174382 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istiod-ca-cert\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.179315 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.179294 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-credential-socket\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.179461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.179445 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-data\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:39.198678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.174834 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istiod-ca-cert" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:39.209103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.209069 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-credential-socket" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.226177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.210692 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-data" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:39.262678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.261938 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m" Nov 01 22:55:39.297010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.296968 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:55:39.297235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.297025 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm"] Nov 01 22:55:39.298592 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.298555 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" containerName="payload-processing" Nov 01 22:55:39.298726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.298595 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" containerName="payload-processing" Nov 01 22:55:39.298726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.298644 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ca59b420-9ae9-4196-9cb4-c01f68908a50" containerName="payload-pre-processing" Nov 01 22:55:39.298726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.298652 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca59b420-9ae9-4196-9cb4-c01f68908a50" containerName="payload-pre-processing" Nov 01 22:55:39.298967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.298952 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" containerName="payload-processing" Nov 01 22:55:39.299053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.298975 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" containerName="istio-proxy" Nov 01 22:55:39.299053 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.298988 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca59b420-9ae9-4196-9cb4-c01f68908a50" containerName="payload-pre-processing" Nov 01 22:55:39.402638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.395341 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:39.434650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.420071 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-podinfo" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:55:39.434650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.420304 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-kube-api-access-b8kjx" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "kube-api-access-b8kjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:39.434650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.422049 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.434650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.422209 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.434650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.422275 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.434650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.422328 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.434650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.422499 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.436591 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.468701 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.470511 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.470858 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.471125 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.489541 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2" event={"ID":"b1489db7-ad64-4845-8fdc-2474a55aaf0e","Type":"ContainerDied","Data":"7801be417e86792e6aa272cafa34dee2903dcb71ac1f02ec36b4a8520573a649"} Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.489621 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm"] Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.489737 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.505495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.503763 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:39.544731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.528973 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:55:39.547724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.547661 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:55:39.562943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.562898 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:39.582999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.572891 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:55:39.583471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.573200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:39.583777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.579774 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca59b420-9ae9-4196-9cb4-c01f68908a50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca59b420-9ae9-4196-9cb4-c01f68908a50-plugins-config-volume\") pod \"ca59b420-9ae9-4196-9cb4-c01f68908a50\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " Nov 01 22:55:39.584316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.584301 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-kube-api-access-dwjql\" (UniqueName: \"kubernetes.io/projected/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-kube-api-access-dwjql\") pod \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " Nov 01 22:55:39.584520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.584506 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca59b420-9ae9-4196-9cb4-c01f68908a50-kube-api-access-9hd5r\" (UniqueName: \"kubernetes.io/projected/ca59b420-9ae9-4196-9cb4-c01f68908a50-kube-api-access-9hd5r\") pod \"ca59b420-9ae9-4196-9cb4-c01f68908a50\" (UID: \"ca59b420-9ae9-4196-9cb4-c01f68908a50\") " Nov 01 22:55:39.584762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.584749 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-plugins-config-volume\") pod \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\" (UID: \"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2\") " Nov 01 22:55:39.586840 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.586804 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.587115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.587083 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5s9x\" (UniqueName: \"kubernetes.io/projected/2d879430-b569-4893-bbdd-265ba2b8292c-kube-api-access-f5s9x\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.638155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.638123 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.642682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.642651 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.683157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.683116 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.683836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.683809 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvmz\" (UniqueName: \"kubernetes.io/projected/4cdbd6c1-921c-4186-88da-42a71ca1f279-kube-api-access-5tvmz\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.685418 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.685366 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8kjx\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-kube-api-access-b8kjx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.687370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.687335 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.687500 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.592020 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:55:39.687740 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.687727 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:55:39.700778 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.700744 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" event={"ID":"ca59b420-9ae9-4196-9cb4-c01f68908a50","Type":"ContainerDied","Data":"a950c8fb96b08e416668008ecac7dc565708bf0ca5957bafddf0e8af91533e4d"} Nov 01 22:55:39.701056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.701040 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:55:39.701281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.701269 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:55:39.701420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.701408 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:55:39.701558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.660699 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj" Nov 01 22:55:39.702025 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.611522 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-kube-api-access-dwjql" pod "7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" (UID: "7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2"). InnerVolumeSpecName "kube-api-access-dwjql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:39.702168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.627358 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca59b420-9ae9-4196-9cb4-c01f68908a50-plugins-config-volume" pod "ca59b420-9ae9-4196-9cb4-c01f68908a50" (UID: "ca59b420-9ae9-4196-9cb4-c01f68908a50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:39.702276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.627561 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca59b420-9ae9-4196-9cb4-c01f68908a50-kube-api-access-9hd5r" pod "ca59b420-9ae9-4196-9cb4-c01f68908a50" (UID: "ca59b420-9ae9-4196-9cb4-c01f68908a50"). InnerVolumeSpecName "kube-api-access-9hd5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:39.702382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.643641 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-plugins-config-volume" pod "7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" (UID: "7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:39.702492 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.660627 4981 generic.go:356] "Generic (PLEG): container finished" podID="ca59b420-9ae9-4196-9cb4-c01f68908a50" containerID="a950c8fb96b08e416668008ecac7dc565708bf0ca5957bafddf0e8af91533e4d" exitCode=0 Nov 01 22:55:39.702746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.696465 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:55:39.705947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.705931 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:55:39.709764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.709729 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:39.714319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.714303 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:55:39.802602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.800922 4981 scope.go:122] "RemoveContainer" containerID="cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463" Nov 01 22:55:39.802602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.801136 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:55:39.823994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.823956 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.824236 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.824216 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.835595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.824416 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.836082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.836049 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvmz\" (UniqueName: \"kubernetes.io/projected/4cdbd6c1-921c-4186-88da-42a71ca1f279-kube-api-access-5tvmz\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.836511 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.836485 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.836748 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.836730 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5s9x\" (UniqueName: \"kubernetes.io/projected/2d879430-b569-4893-bbdd-265ba2b8292c-kube-api-access-f5s9x\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.837139 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.837118 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca59b420-9ae9-4196-9cb4-c01f68908a50-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.837283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.837258 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjql\" (UniqueName: \"kubernetes.io/projected/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-kube-api-access-dwjql\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.837404 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.837387 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hd5r\" (UniqueName: \"kubernetes.io/projected/ca59b420-9ae9-4196-9cb4-c01f68908a50-kube-api-access-9hd5r\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.837503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.837488 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:39.898369 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.898330 4981 generic.go:356] "Generic (PLEG): container finished" podID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" containerID="c0361a14518e2227e9c8ff497a12bcedd1582a9677d4197c96e506963fbd3ca8" exitCode=0 Nov 01 22:55:39.900115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.900088 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" Nov 01 22:55:39.921631 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.912227 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs" event={"ID":"7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2","Type":"ContainerDied","Data":"c0361a14518e2227e9c8ff497a12bcedd1582a9677d4197c96e506963fbd3ca8"} Nov 01 22:55:39.969684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.923395 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.969684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.941248 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvmz\" (UniqueName: \"kubernetes.io/projected/4cdbd6c1-921c-4186-88da-42a71ca1f279-kube-api-access-5tvmz\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.969684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.942623 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:55:39.972406 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.972356 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:39.974896 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.974878 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:55:39.979809 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.979789 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:39.980076 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.980060 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:39.980801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.980786 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:39.981696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.981681 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:39.983192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.983163 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:39.990933 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:39.990826 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 22:55:40.004452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.004409 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 22:55:40.020629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.017750 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:40.020629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.017823 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:40.029136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.024061 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:40.029136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.024120 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:40.029136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.028235 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 22:55:40.029136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.029065 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:40.032999 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.032972 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5s9x\" (UniqueName: \"kubernetes.io/projected/2d879430-b569-4893-bbdd-265ba2b8292c-kube-api-access-f5s9x\") pod \"maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:40.037729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.035726 4981 scope.go:122] "RemoveContainer" containerID="2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7" Nov 01 22:55:40.037729 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:40.037653 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7\": container with ID starting with 2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7 not found: ID does not exist" containerID="2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7" Nov 01 22:55:40.037729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.037686 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7"} err="failed to get container status \"2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7\": rpc error: code = NotFound desc = could not find container \"2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7\": container with ID starting with 2f3f5fd67ab9f12a5b37fc7b9e116f47b8eee450c2f1a01d943d4b5e5fd67db7 not found: ID does not exist" Nov 01 22:55:40.038469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.038016 4981 scope.go:122] "RemoveContainer" containerID="cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463" Nov 01 22:55:40.038955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.038635 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:40.038955 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:40.038749 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463\": container with ID starting with cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463 not found: ID does not exist" containerID="cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463" Nov 01 22:55:40.038955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.038784 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463"} err="failed to get container status \"cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463\": rpc error: code = NotFound desc = could not find container \"cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463\": container with ID starting with cdc84ab3b7d8a0caa58c0c39948428cd33ba1fa8284ca408049715bae9e4e463 not found: ID does not exist" Nov 01 22:55:40.038955 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.038807 4981 scope.go:122] "RemoveContainer" containerID="a950c8fb96b08e416668008ecac7dc565708bf0ca5957bafddf0e8af91533e4d" Nov 01 22:55:40.040037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.039315 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:55:40.077329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.077305 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:55:40.092725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.092691 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:40.093337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.092915 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:55:40.093499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.093468 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:40.125668 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.122315 4981 scope.go:122] "RemoveContainer" containerID="c0361a14518e2227e9c8ff497a12bcedd1582a9677d4197c96e506963fbd3ca8" Nov 01 22:55:40.127771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.127490 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:40.127771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.127683 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\") pod \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\" (UID: \"b1489db7-ad64-4845-8fdc-2474a55aaf0e\") " Nov 01 22:55:40.132695 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:40.132652 4981 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/b1489db7-ad64-4845-8fdc-2474a55aaf0e/volumes/kubernetes.io~projected/istio-token Nov 01 22:55:40.134237 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.134214 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:40.136608 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:40.134105 4981 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/b1489db7-ad64-4845-8fdc-2474a55aaf0e/volumes/kubernetes.io~empty-dir/istio-envoy Nov 01 22:55:40.136766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.136736 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy" pod "b1489db7-ad64-4845-8fdc-2474a55aaf0e" (UID: "b1489db7-ad64-4845-8fdc-2474a55aaf0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:55:40.137398 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.137376 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:40.137538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.137522 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1489db7-ad64-4845-8fdc-2474a55aaf0e-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:40.193347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.193315 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:40.219245 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:40.216792 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca59b420_9ae9_4196_9cb4_c01f68908a50.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0ae1bb_f3ae_4f70_b9a0_1b0832cea7a2.slice\": RecentStats: unable to find data in memory cache]" Nov 01 22:55:40.412633 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.412595 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj"] Nov 01 22:55:40.430206 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.430179 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44a9aa-669659ff7f-h64zj"] Nov 01 22:55:40.438462 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.438409 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs"] Nov 01 22:55:40.448481 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.448453 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44a9aa-76989cc44c-g8szs"] Nov 01 22:55:40.451322 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:40.451295 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdbd6c1_921c_4186_88da_42a71ca1f279.slice/crio-cd2051ca63c8ded5556203c96d21508c43a4c90b12c87a5f7000a0c15c04d537 WatchSource:0}: Error finding container cd2051ca63c8ded5556203c96d21508c43a4c90b12c87a5f7000a0c15c04d537: Status 404 returned error can't find the container with id cd2051ca63c8ded5556203c96d21508c43a4c90b12c87a5f7000a0c15c04d537 Nov 01 22:55:40.459333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.457437 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m"] Nov 01 22:55:40.565103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.565068 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm"] Nov 01 22:55:40.614826 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.614787 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:40.689020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.688988 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2"] Nov 01 22:55:40.694512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.694481 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-44a9aa-openshift-default-9b89654b5-654x2"] Nov 01 22:55:40.741095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.741057 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2" path="/var/lib/kubelet/pods/7b0ae1bb-f3ae-4f70-b9a0-1b0832cea7a2/volumes" Nov 01 22:55:40.741830 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.741809 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1489db7-ad64-4845-8fdc-2474a55aaf0e" path="/var/lib/kubelet/pods/b1489db7-ad64-4845-8fdc-2474a55aaf0e/volumes" Nov 01 22:55:40.742280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.742266 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca59b420-9ae9-4196-9cb4-c01f68908a50" path="/var/lib/kubelet/pods/ca59b420-9ae9-4196-9cb4-c01f68908a50/volumes" Nov 01 22:55:40.908916 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.908884 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" event={"ID":"4cdbd6c1-921c-4186-88da-42a71ca1f279","Type":"ContainerStarted","Data":"cd2051ca63c8ded5556203c96d21508c43a4c90b12c87a5f7000a0c15c04d537"} Nov 01 22:55:40.913536 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:40.913509 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" event={"ID":"2d879430-b569-4893-bbdd-265ba2b8292c","Type":"ContainerStarted","Data":"45aeadbdcc17fd5152d1760680be84cb4e94f897e302d1c5bf2a6bb0cbedcb86"} Nov 01 22:55:41.453001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:41.452847 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:41.565031 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:41.564971 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:41.953593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:41.953548 4981 generic.go:356] "Generic (PLEG): container finished" podID="2d879430-b569-4893-bbdd-265ba2b8292c" containerID="74c344857776e8e21494f798a0fe2222b232599e5387a2bb0a2f9c7d9e1db4a5" exitCode=0 Nov 01 22:55:41.954041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:41.953892 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" event={"ID":"2d879430-b569-4893-bbdd-265ba2b8292c","Type":"ContainerDied","Data":"74c344857776e8e21494f798a0fe2222b232599e5387a2bb0a2f9c7d9e1db4a5"} Nov 01 22:55:41.956215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:41.956191 4981 generic.go:356] "Generic (PLEG): container finished" podID="4cdbd6c1-921c-4186-88da-42a71ca1f279" containerID="cb9ebd4908b83e6d790d1a03deb99ffded0f6ab514b8d99812fa4358e507deb0" exitCode=0 Nov 01 22:55:41.956669 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:41.956639 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" event={"ID":"4cdbd6c1-921c-4186-88da-42a71ca1f279","Type":"ContainerDied","Data":"cb9ebd4908b83e6d790d1a03deb99ffded0f6ab514b8d99812fa4358e507deb0"} Nov 01 22:55:42.244374 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:42.244295 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:42.954470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:42.954433 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:43.274071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.274043 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:43.277388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.277356 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:43.336399 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.336361 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d879430-b569-4893-bbdd-265ba2b8292c-kube-api-access-f5s9x\" (UniqueName: \"kubernetes.io/projected/2d879430-b569-4893-bbdd-265ba2b8292c-kube-api-access-f5s9x\") pod \"2d879430-b569-4893-bbdd-265ba2b8292c\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " Nov 01 22:55:43.336677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.336637 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-openshift-ca\") pod \"4cdbd6c1-921c-4186-88da-42a71ca1f279\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " Nov 01 22:55:43.336777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.336720 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-xks-ca\") pod \"2d879430-b569-4893-bbdd-265ba2b8292c\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " Nov 01 22:55:43.336902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.336788 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cdbd6c1-921c-4186-88da-42a71ca1f279-kube-api-access-5tvmz\" (UniqueName: \"kubernetes.io/projected/4cdbd6c1-921c-4186-88da-42a71ca1f279-kube-api-access-5tvmz\") pod \"4cdbd6c1-921c-4186-88da-42a71ca1f279\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " Nov 01 22:55:43.336902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.336827 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-openshift-ca\") pod \"2d879430-b569-4893-bbdd-265ba2b8292c\" (UID: \"2d879430-b569-4893-bbdd-265ba2b8292c\") " Nov 01 22:55:43.337136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.337100 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-xks-ca\") pod \"4cdbd6c1-921c-4186-88da-42a71ca1f279\" (UID: \"4cdbd6c1-921c-4186-88da-42a71ca1f279\") " Nov 01 22:55:43.338551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.338502 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-openshift-ca" pod "2d879430-b569-4893-bbdd-265ba2b8292c" (UID: "2d879430-b569-4893-bbdd-265ba2b8292c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:43.339064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.339035 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-openshift-ca" pod "4cdbd6c1-921c-4186-88da-42a71ca1f279" (UID: "4cdbd6c1-921c-4186-88da-42a71ca1f279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:43.340234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.340195 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d879430-b569-4893-bbdd-265ba2b8292c-kube-api-access-f5s9x" pod "2d879430-b569-4893-bbdd-265ba2b8292c" (UID: "2d879430-b569-4893-bbdd-265ba2b8292c"). InnerVolumeSpecName "kube-api-access-f5s9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:43.340432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.340411 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdbd6c1-921c-4186-88da-42a71ca1f279-kube-api-access-5tvmz" pod "4cdbd6c1-921c-4186-88da-42a71ca1f279" (UID: "4cdbd6c1-921c-4186-88da-42a71ca1f279"). InnerVolumeSpecName "kube-api-access-5tvmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:43.341119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.341087 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-xks-ca" pod "2d879430-b569-4893-bbdd-265ba2b8292c" (UID: "2d879430-b569-4893-bbdd-265ba2b8292c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:43.341302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.341275 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-xks-ca" pod "4cdbd6c1-921c-4186-88da-42a71ca1f279" (UID: "4cdbd6c1-921c-4186-88da-42a71ca1f279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:55:43.439974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.439940 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5s9x\" (UniqueName: \"kubernetes.io/projected/2d879430-b569-4893-bbdd-265ba2b8292c-kube-api-access-f5s9x\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:43.439974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.439967 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:43.439974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.439977 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:43.439974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.439986 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tvmz\" (UniqueName: \"kubernetes.io/projected/4cdbd6c1-921c-4186-88da-42a71ca1f279-kube-api-access-5tvmz\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:43.440333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.439995 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d879430-b569-4893-bbdd-265ba2b8292c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:43.440333 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.440004 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cdbd6c1-921c-4186-88da-42a71ca1f279-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:43.877110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.877080 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:43.979720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.979690 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" Nov 01 22:55:43.980142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.979716 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m" event={"ID":"4cdbd6c1-921c-4186-88da-42a71ca1f279","Type":"ContainerDied","Data":"cd2051ca63c8ded5556203c96d21508c43a4c90b12c87a5f7000a0c15c04d537"} Nov 01 22:55:43.980142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.979767 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2051ca63c8ded5556203c96d21508c43a4c90b12c87a5f7000a0c15c04d537" Nov 01 22:55:43.981465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.981446 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" Nov 01 22:55:43.981593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.981486 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm" event={"ID":"2d879430-b569-4893-bbdd-265ba2b8292c","Type":"ContainerDied","Data":"45aeadbdcc17fd5152d1760680be84cb4e94f897e302d1c5bf2a6bb0cbedcb86"} Nov 01 22:55:43.981593 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:43.981524 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45aeadbdcc17fd5152d1760680be84cb4e94f897e302d1c5bf2a6bb0cbedcb86" Nov 01 22:55:44.528742 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:44.528704 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:55:44.566927 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:44.566893 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:45.234517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:45.234477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:45.973208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:45.973172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:46.777876 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:46.777819 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:47.498995 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:47.498966 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:47.766032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:47.765951 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc"] Nov 01 22:55:47.834351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:47.834314 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:47.938699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:47.938664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:48.621739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:48.621706 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:49.296426 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:49.296394 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:50.064350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:50.064314 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:50.742920 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:50.742889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:51.541484 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:51.541450 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:52.233387 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:52.233357 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:52.766848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:52.766795 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" podUID="a1a1edad-7830-497a-9de1-bc320ed49297" containerName="payload-processing" containerID="cri-o://4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4" gracePeriod=25 Nov 01 22:55:52.941571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:52.941531 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:53.140900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.140876 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:55:53.165709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.165676 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a1edad-7830-497a-9de1-bc320ed49297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a1edad-7830-497a-9de1-bc320ed49297-plugins-config-volume\") pod \"a1a1edad-7830-497a-9de1-bc320ed49297\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " Nov 01 22:55:53.165880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.165773 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a1edad-7830-497a-9de1-bc320ed49297-kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/a1a1edad-7830-497a-9de1-bc320ed49297-kube-api-access-4vlvg\") pod \"a1a1edad-7830-497a-9de1-bc320ed49297\" (UID: \"a1a1edad-7830-497a-9de1-bc320ed49297\") " Nov 01 22:55:53.166131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.166103 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a1edad-7830-497a-9de1-bc320ed49297-plugins-config-volume" pod "a1a1edad-7830-497a-9de1-bc320ed49297" (UID: "a1a1edad-7830-497a-9de1-bc320ed49297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:55:53.166789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.166765 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a1edad-7830-497a-9de1-bc320ed49297-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:53.168379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.168352 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a1edad-7830-497a-9de1-bc320ed49297-kube-api-access-4vlvg" pod "a1a1edad-7830-497a-9de1-bc320ed49297" (UID: "a1a1edad-7830-497a-9de1-bc320ed49297"). InnerVolumeSpecName "kube-api-access-4vlvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:55:53.231431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.231394 4981 generic.go:356] "Generic (PLEG): container finished" podID="a1a1edad-7830-497a-9de1-bc320ed49297" containerID="4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4" exitCode=0 Nov 01 22:55:53.231675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.231433 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" event={"ID":"a1a1edad-7830-497a-9de1-bc320ed49297","Type":"ContainerDied","Data":"4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4"} Nov 01 22:55:53.231675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.231515 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" event={"ID":"a1a1edad-7830-497a-9de1-bc320ed49297","Type":"ContainerDied","Data":"f20431be9ac899d7fe988236d512b8b0fb15f35c267d85af8d1e3ec856a42530"} Nov 01 22:55:53.231675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.231440 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" Nov 01 22:55:53.231675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.231549 4981 scope.go:122] "RemoveContainer" containerID="4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4" Nov 01 22:55:53.247753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.247658 4981 scope.go:122] "RemoveContainer" containerID="4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4" Nov 01 22:55:53.250690 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:55:53.250666 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4\": container with ID starting with 4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4 not found: ID does not exist" containerID="4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4" Nov 01 22:55:53.250815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.250696 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4"} err="failed to get container status \"4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4\": rpc error: code = NotFound desc = could not find container \"4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4\": container with ID starting with 4f0eed8ff3ca1110efc669009a0aa57d77229eaadb0638775e4633b2d8a084b4 not found: ID does not exist" Nov 01 22:55:53.260629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.260562 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc"] Nov 01 22:55:53.266066 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.266046 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc"] Nov 01 22:55:53.268490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.268473 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vlvg\" (UniqueName: \"kubernetes.io/projected/a1a1edad-7830-497a-9de1-bc320ed49297-kube-api-access-4vlvg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:55:53.668782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.668751 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:53.772124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:53.772085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:54.084629 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:54.084571 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-58c957-fdb69dd59-qrzhc" podUID="a1a1edad-7830-497a-9de1-bc320ed49297" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.200:50052\" within 1s: context deadline exceeded" Nov 01 22:55:54.084829 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:55:54.084678 4981 logging.go:55] [core] [Channel #7397 SubChannel #7398]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.200:50052", ServerName: "10.134.0.200:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.200:50052: operation was canceled" Nov 01 22:55:54.275792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:54.275752 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:54.606039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:54.604693 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:54.702584 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:54.702537 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:54.738867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:54.738833 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a1edad-7830-497a-9de1-bc320ed49297" path="/var/lib/kubelet/pods/a1a1edad-7830-497a-9de1-bc320ed49297/volumes" Nov 01 22:55:55.507697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:55.505435 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:56.385256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:56.385215 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:56.497616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:56.497551 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:56.598674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:56.598638 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:57.265679 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:57.265645 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:57.961886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:57.961844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:58.758988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:58.758958 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:55:59.416451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:55:59.416416 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:00.066588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:00.066548 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:00.758508 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:00.758479 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:01.043415 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:01.043335 4981 scope.go:122] "RemoveContainer" containerID="46435f6c797d0f90b4c60d8cc6b1ba383402700e57e4193e8e03549ed4dea02a" Nov 01 22:56:01.055077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:01.055058 4981 scope.go:122] "RemoveContainer" containerID="275819ea563b9e380c0216b96bca273602e0c875cf547863a4f19a91f6b5f583" Nov 01 22:56:01.390255 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:01.390151 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:02.079954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:02.079913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:02.800885 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:02.800844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:02.916598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:02.903530 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:03.505702 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:03.505657 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:04.178566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:04.178533 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:04.965970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:04.965937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:05.571627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:05.571559 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:06.197846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:06.197804 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:07.015215 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:07.015181 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:07.574548 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:07.574496 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:07.742734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:07.739084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:07.905695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:07.901016 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:08.061627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:08.057187 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm"] Nov 01 22:56:08.091427 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:08.091373 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-58c957-jzpzm"] Nov 01 22:56:08.104940 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:08.104759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:08.427829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:08.427793 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:08.737811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:08.737736 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d879430-b569-4893-bbdd-265ba2b8292c" path="/var/lib/kubelet/pods/2d879430-b569-4893-bbdd-265ba2b8292c/volumes" Nov 01 22:56:09.150528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:09.150417 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:10.067771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:10.067733 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:10.857810 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:10.857771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:12.364724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.364675 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.383911 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.383362 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 01 22:56:12.414720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.414674 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:56:12.481213 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.481169 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.502925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.501701 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.530849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.529875 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.593833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.593786 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.625819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.625712 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.627327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.627291 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.777677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.777638 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.866710 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.866664 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.885755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.885656 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.886982 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.886950 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:12.905724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:12.905679 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.088960 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.088924 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:13.209016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.208800 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.280177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.279937 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 01 22:56:13.369760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.367248 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.369760 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.367711 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:56:13.388617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.383909 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.388617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.384338 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:56:13.424485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.422677 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:56:13.424485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.423351 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:56:13.489335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.485672 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:56:13.489335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.486517 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:56:13.523210 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.523160 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:56:13.547697 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.547656 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.735726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.678544 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:56:13.735726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.696874 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 01 22:56:13.735726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.697224 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:56:13.802659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.794813 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:13.802659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.795237 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 01 22:56:13.906825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.906748 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.113093 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:13.967347 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.123186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.038849 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.203917 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204086 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204128 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204164 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204199 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204234 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204269 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204306 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204344 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.204378 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:56:14.222875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.209672 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 01 22:56:14.227610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.224655 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:56:14.361609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.361548 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:56:14.378833 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.368828 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.404600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.395723 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.404600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.404081 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:56:14.433628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.429620 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.529477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.529433 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 01 22:56:14.546811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.546769 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.555983 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.555947 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 01 22:56:14.697064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.697024 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.697466 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.697435 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.887609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.726071 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 22:56:14.896428 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.729545 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.897569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.759609 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:14.903729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.797687 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:14.929650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:14.926970 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:15.020625 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.020523 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:15.031783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.028431 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded" Nov 01 22:56:15.045601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.036615 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:15.109465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.109425 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:15.130606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.130551 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Nov 01 22:56:15.368980 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.368869 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 01 22:56:15.420755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.402919 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:15.549677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.549515 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 01 22:56:16.170005 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.169964 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 01 22:56:16.201271 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.697478 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:16.201607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.790890 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 01 22:56:16.201850 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:56:15.893990 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:56:16.202017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.170874 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Nov 01 22:56:16.202205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.171250 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 01 22:56:16.202355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.171300 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" podUID="e7f7e966-fddb-4b4c-b76e-f2c2f55c8350" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded" Nov 01 22:56:16.202449 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.200073 4981 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:16.202543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.200693 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:16.202996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.202964 4981 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-n87cr container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.7:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 22:56:16.203140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.203122 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-n87cr" podUID="8deca68d-1a0b-4841-9c91-0682262d59c7" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.7:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:16.204435 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:15.595631 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:16.242203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.241678 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 01 22:56:16.246364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.244625 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:16.256201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.256169 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Nov 01 22:56:16.293322 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:56:16.293290 4981 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.405s" Nov 01 22:56:16.293525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.293359 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 22:56:16.293525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.293389 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 22:56:16.293525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.293413 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 22:56:16.293525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.293491 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 22:56:16.295209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.295179 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 22:56:16.296471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.296381 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 22:56:16.296471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.296430 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8" Nov 01 22:56:16.296471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.296459 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 22:56:16.298248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.298230 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 22:56:16.303439 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.303419 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 22:56:16.306022 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.306001 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 22:56:16.308489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.308460 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 22:56:16.315931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.315907 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 22:56:16.324013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.323994 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 22:56:16.386184 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.386147 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m"] Nov 01 22:56:16.391161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.391126 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e202e7-xx56m"] Nov 01 22:56:16.420185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.420087 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:16.612312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.612270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:16.739208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.739177 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cdbd6c1-921c-4186-88da-42a71ca1f279" path="/var/lib/kubelet/pods/4cdbd6c1-921c-4186-88da-42a71ca1f279/volumes" Nov 01 22:56:16.851263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.851232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:16.956408 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:16.956376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:17.206881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.206852 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:17.897129 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.897089 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s"] Nov 01 22:56:17.898410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898386 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a1a1edad-7830-497a-9de1-bc320ed49297" containerName="payload-processing" Nov 01 22:56:17.898410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898408 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a1edad-7830-497a-9de1-bc320ed49297" containerName="payload-processing" Nov 01 22:56:17.898568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898424 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2d879430-b569-4893-bbdd-265ba2b8292c" containerName="revoke-keys" Nov 01 22:56:17.898568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898429 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d879430-b569-4893-bbdd-265ba2b8292c" containerName="revoke-keys" Nov 01 22:56:17.898568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898514 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4cdbd6c1-921c-4186-88da-42a71ca1f279" containerName="revoke-keys" Nov 01 22:56:17.898568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898522 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdbd6c1-921c-4186-88da-42a71ca1f279" containerName="revoke-keys" Nov 01 22:56:17.898758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898743 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a1edad-7830-497a-9de1-bc320ed49297" containerName="payload-processing" Nov 01 22:56:17.898793 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898766 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d879430-b569-4893-bbdd-265ba2b8292c" containerName="revoke-keys" Nov 01 22:56:17.898827 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.898797 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cdbd6c1-921c-4186-88da-42a71ca1f279" containerName="revoke-keys" Nov 01 22:56:17.903380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.903325 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:17.908882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.908859 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s"] Nov 01 22:56:17.928134 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.928101 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:56:17.928264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:17.928152 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:56:18.028806 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.028772 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.028979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.028817 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.029030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.028994 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhhj\" (UniqueName: \"kubernetes.io/projected/31acdd55-098c-4a00-850d-6dbe02f7638a-kube-api-access-jrhhj\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.063194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.063162 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:18.158869 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.158758 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhhj\" (UniqueName: \"kubernetes.io/projected/31acdd55-098c-4a00-850d-6dbe02f7638a-kube-api-access-jrhhj\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.159278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.159246 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.159467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.159300 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.160346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.160320 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.163311 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.163282 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.169194 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.169170 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhhj\" (UniqueName: \"kubernetes.io/projected/31acdd55-098c-4a00-850d-6dbe02f7638a-kube-api-access-jrhhj\") pod \"maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.222746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.222716 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:18.513780 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:56:18.513656 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31acdd55_098c_4a00_850d_6dbe02f7638a.slice/crio-78f0de00d5a6f1be319b93b435ef4ce1d56b134d820ecd1606f68808942c83bc WatchSource:0}: Error finding container 78f0de00d5a6f1be319b93b435ef4ce1d56b134d820ecd1606f68808942c83bc: Status 404 returned error can't find the container with id 78f0de00d5a6f1be319b93b435ef4ce1d56b134d820ecd1606f68808942c83bc Nov 01 22:56:18.515488 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.515465 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s"] Nov 01 22:56:18.981292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:18.981255 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:19.205285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:19.205244 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" event={"ID":"31acdd55-098c-4a00-850d-6dbe02f7638a","Type":"ContainerStarted","Data":"ab9ca7d9dc39474541c04a5894133198ddce35a706cbc345ba87c943bb9f9ef1"} Nov 01 22:56:19.205495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:19.205445 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" event={"ID":"31acdd55-098c-4a00-850d-6dbe02f7638a","Type":"ContainerStarted","Data":"78f0de00d5a6f1be319b93b435ef4ce1d56b134d820ecd1606f68808942c83bc"} Nov 01 22:56:19.229397 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:19.229349 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" podStartSLOduration=2.229333297 podStartE2EDuration="2.229333297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:19.2269408 +0000 UTC m=+1944.654818731" watchObservedRunningTime="2026-11-01 22:56:19.229333297 +0000 UTC m=+1944.657211229" Nov 01 22:56:19.669525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:19.669480 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:20.217373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:20.217337 4981 generic.go:356] "Generic (PLEG): container finished" podID="31acdd55-098c-4a00-850d-6dbe02f7638a" containerID="ab9ca7d9dc39474541c04a5894133198ddce35a706cbc345ba87c943bb9f9ef1" exitCode=0 Nov 01 22:56:20.217871 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:20.217468 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" event={"ID":"31acdd55-098c-4a00-850d-6dbe02f7638a","Type":"ContainerDied","Data":"ab9ca7d9dc39474541c04a5894133198ddce35a706cbc345ba87c943bb9f9ef1"} Nov 01 22:56:20.516707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:20.516626 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:21.246988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.246959 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:21.486635 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.486535 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:21.533252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.533221 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-openshift-ca\") pod \"31acdd55-098c-4a00-850d-6dbe02f7638a\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " Nov 01 22:56:21.533542 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.533527 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-xks-ca\") pod \"31acdd55-098c-4a00-850d-6dbe02f7638a\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " Nov 01 22:56:21.533664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.533562 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31acdd55-098c-4a00-850d-6dbe02f7638a-kube-api-access-jrhhj\" (UniqueName: \"kubernetes.io/projected/31acdd55-098c-4a00-850d-6dbe02f7638a-kube-api-access-jrhhj\") pod \"31acdd55-098c-4a00-850d-6dbe02f7638a\" (UID: \"31acdd55-098c-4a00-850d-6dbe02f7638a\") " Nov 01 22:56:21.533664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.533641 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-openshift-ca" pod "31acdd55-098c-4a00-850d-6dbe02f7638a" (UID: "31acdd55-098c-4a00-850d-6dbe02f7638a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:21.534411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.534390 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:21.535848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.535818 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-xks-ca" pod "31acdd55-098c-4a00-850d-6dbe02f7638a" (UID: "31acdd55-098c-4a00-850d-6dbe02f7638a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:21.535848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.535829 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31acdd55-098c-4a00-850d-6dbe02f7638a-kube-api-access-jrhhj" pod "31acdd55-098c-4a00-850d-6dbe02f7638a" (UID: "31acdd55-098c-4a00-850d-6dbe02f7638a"). InnerVolumeSpecName "kube-api-access-jrhhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:21.644489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.644448 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31acdd55-098c-4a00-850d-6dbe02f7638a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:21.644949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:21.644490 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhhj\" (UniqueName: \"kubernetes.io/projected/31acdd55-098c-4a00-850d-6dbe02f7638a-kube-api-access-jrhhj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:22.246199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:22.246172 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" Nov 01 22:56:22.246199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:22.246163 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s" event={"ID":"31acdd55-098c-4a00-850d-6dbe02f7638a","Type":"ContainerDied","Data":"78f0de00d5a6f1be319b93b435ef4ce1d56b134d820ecd1606f68808942c83bc"} Nov 01 22:56:22.246452 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:22.246242 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78f0de00d5a6f1be319b93b435ef4ce1d56b134d820ecd1606f68808942c83bc" Nov 01 22:56:22.440571 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:22.437787 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:22.949283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:22.949256 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:23.053148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.052641 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:23.254282 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.254135 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:23.260295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.260264 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8"] Nov 01 22:56:23.260538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.260515 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" containerID="cri-o://4263a5d250ec163e44029a2c30fdb28464cbdd62fca1bc21ddbef6f563920a2c" gracePeriod=30 Nov 01 22:56:23.367424 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.367388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:23.681937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.681904 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:23.721968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.721932 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt"] Nov 01 22:56:23.723001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.722981 4981 cpu_manager.go:414] "containerMap: removing container" podUID="31acdd55-098c-4a00-850d-6dbe02f7638a" containerName="revoke-keys" Nov 01 22:56:23.723084 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.723002 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31acdd55-098c-4a00-850d-6dbe02f7638a" containerName="revoke-keys" Nov 01 22:56:23.723223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.723211 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="31acdd55-098c-4a00-850d-6dbe02f7638a" containerName="revoke-keys" Nov 01 22:56:23.727506 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.727482 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.740888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.740405 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt"] Nov 01 22:56:23.789525 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.789465 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:56:23.789752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.789729 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:56:23.821020 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.820980 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgvc\" (UniqueName: \"kubernetes.io/projected/0dd42122-3520-40fd-a632-6b28769b6730-kube-api-access-rxgvc\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.821212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.821138 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.821612 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.821561 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.923841 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.923803 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgvc\" (UniqueName: \"kubernetes.io/projected/0dd42122-3520-40fd-a632-6b28769b6730-kube-api-access-rxgvc\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.924011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.923889 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.924101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.924089 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.924728 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.924709 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.926256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.926236 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:23.932954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:23.932892 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgvc\" (UniqueName: \"kubernetes.io/projected/0dd42122-3520-40fd-a632-6b28769b6730-kube-api-access-rxgvc\") pod \"maas-api-revoke-keys-e2e-inf-b-f07902-gfppt\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:24.047281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:24.047241 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:24.110969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:24.110931 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:24.208937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:24.208845 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:24.552627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:24.551838 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:24.565599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:24.561983 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt"] Nov 01 22:56:24.625598 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:56:24.624446 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd42122_3520_40fd_a632_6b28769b6730.slice/crio-a10540037168faa7e8d71024a2e93a79d7ffab2bdb29480bd0cb30c003c3594e WatchSource:0}: Error finding container a10540037168faa7e8d71024a2e93a79d7ffab2bdb29480bd0cb30c003c3594e: Status 404 returned error can't find the container with id a10540037168faa7e8d71024a2e93a79d7ffab2bdb29480bd0cb30c003c3594e Nov 01 22:56:25.472252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:25.472210 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:25.572610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:25.568262 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" event={"ID":"0dd42122-3520-40fd-a632-6b28769b6730","Type":"ContainerStarted","Data":"a3b2236310aa1469bdda2fb00545fbddd54de2ae87a6150fabb082b09199be5a"} Nov 01 22:56:25.572610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:25.568335 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" event={"ID":"0dd42122-3520-40fd-a632-6b28769b6730","Type":"ContainerStarted","Data":"a10540037168faa7e8d71024a2e93a79d7ffab2bdb29480bd0cb30c003c3594e"} Nov 01 22:56:25.602998 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:25.602921 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" podStartSLOduration=2.602899844 podStartE2EDuration="2.602899844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:25.591533595 +0000 UTC m=+1951.019411527" watchObservedRunningTime="2026-11-01 22:56:25.602899844 +0000 UTC m=+1951.030777776" Nov 01 22:56:26.159221 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:26.159144 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:26.579936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:26.579902 4981 generic.go:356] "Generic (PLEG): container finished" podID="0dd42122-3520-40fd-a632-6b28769b6730" containerID="a3b2236310aa1469bdda2fb00545fbddd54de2ae87a6150fabb082b09199be5a" exitCode=0 Nov 01 22:56:26.580534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:26.579985 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" event={"ID":"0dd42122-3520-40fd-a632-6b28769b6730","Type":"ContainerDied","Data":"a3b2236310aa1469bdda2fb00545fbddd54de2ae87a6150fabb082b09199be5a"} Nov 01 22:56:27.125632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.125597 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:27.696146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.696113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:27.795142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.795113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:27.797880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.797860 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:27.817807 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.817776 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-xks-ca\") pod \"0dd42122-3520-40fd-a632-6b28769b6730\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " Nov 01 22:56:27.818123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.818104 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-openshift-ca\") pod \"0dd42122-3520-40fd-a632-6b28769b6730\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " Nov 01 22:56:27.818353 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.818336 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd42122-3520-40fd-a632-6b28769b6730-kube-api-access-rxgvc\" (UniqueName: \"kubernetes.io/projected/0dd42122-3520-40fd-a632-6b28769b6730-kube-api-access-rxgvc\") pod \"0dd42122-3520-40fd-a632-6b28769b6730\" (UID: \"0dd42122-3520-40fd-a632-6b28769b6730\") " Nov 01 22:56:27.820230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.820205 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-openshift-ca" pod "0dd42122-3520-40fd-a632-6b28769b6730" (UID: "0dd42122-3520-40fd-a632-6b28769b6730"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:27.822730 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.822702 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd42122-3520-40fd-a632-6b28769b6730-kube-api-access-rxgvc" pod "0dd42122-3520-40fd-a632-6b28769b6730" (UID: "0dd42122-3520-40fd-a632-6b28769b6730"). InnerVolumeSpecName "kube-api-access-rxgvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:27.822867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.822767 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-xks-ca" pod "0dd42122-3520-40fd-a632-6b28769b6730" (UID: "0dd42122-3520-40fd-a632-6b28769b6730"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:56:27.920913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.920824 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:27.920913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.920857 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgvc\" (UniqueName: \"kubernetes.io/projected/0dd42122-3520-40fd-a632-6b28769b6730-kube-api-access-rxgvc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:27.920913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:27.920872 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd42122-3520-40fd-a632-6b28769b6730-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:28.084775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.084744 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f"] Nov 01 22:56:28.153130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.153091 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:28.255211 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.255178 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:28.729659 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.724911 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:28.819898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.819666 4981 generic.go:356] "Generic (PLEG): container finished" podID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerID="4263a5d250ec163e44029a2c30fdb28464cbdd62fca1bc21ddbef6f563920a2c" exitCode=0 Nov 01 22:56:28.819898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.819842 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" event={"ID":"b153d5e5-f84f-48a3-baea-50a3fbce1b12","Type":"ContainerDied","Data":"4263a5d250ec163e44029a2c30fdb28464cbdd62fca1bc21ddbef6f563920a2c"} Nov 01 22:56:28.820327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.820212 4981 scope.go:122] "RemoveContainer" containerID="3114dd87e4bdcffc34b982431cb285ff3f4c58026bb0a642aea0ac0255ca50a9" Nov 01 22:56:28.826489 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.826457 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" Nov 01 22:56:28.827791 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.826471 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt" event={"ID":"0dd42122-3520-40fd-a632-6b28769b6730","Type":"ContainerDied","Data":"a10540037168faa7e8d71024a2e93a79d7ffab2bdb29480bd0cb30c003c3594e"} Nov 01 22:56:28.828210 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.827825 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10540037168faa7e8d71024a2e93a79d7ffab2bdb29480bd0cb30c003c3594e" Nov 01 22:56:28.985961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:28.985886 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:56:29.010752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.010715 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-socket\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.010986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.010860 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-podinfo\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.011063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.011000 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-kube-api-access-996lr\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-kube-api-access-996lr\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.011063 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.011044 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-credential-socket\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.011169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.011123 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-token\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-token\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.011169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.011135 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-socket" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.011284 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.011186 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-certs\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.011742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.011526 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-credential-socket" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.014034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.012925 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istiod-ca-cert\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.014034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.013135 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-data\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.014034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.013290 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-envoy\") pod \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\" (UID: \"b153d5e5-f84f-48a3-baea-50a3fbce1b12\") " Nov 01 22:56:29.014381 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.014355 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-certs" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.014838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.014813 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istiod-ca-cert" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:29.015520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.015496 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.015687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.015664 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.015687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.015689 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.015978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.015704 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.015978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.015505 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-data" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.019970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.019937 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-podinfo" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:56:29.020429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.020399 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-kube-api-access-996lr" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "kube-api-access-996lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:29.020554 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.020535 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-envoy" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:56:29.021716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.021687 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-token" pod "b153d5e5-f84f-48a3-baea-50a3fbce1b12" (UID: "b153d5e5-f84f-48a3-baea-50a3fbce1b12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:29.067814 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.067755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:29.117788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.117736 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-996lr\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-kube-api-access-996lr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.117788 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.117781 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.118067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.117797 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.118067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.117813 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.118067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.117827 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b153d5e5-f84f-48a3-baea-50a3fbce1b12-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:29.167951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.167914 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:29.838347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.838320 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" Nov 01 22:56:29.838347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.838333 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8" event={"ID":"b153d5e5-f84f-48a3-baea-50a3fbce1b12","Type":"ContainerDied","Data":"89581285b6b3b69b12ce0355684a1a31a0bca5770d9acc9f57c35b479c586411"} Nov 01 22:56:29.838936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.838386 4981 scope.go:122] "RemoveContainer" containerID="4263a5d250ec163e44029a2c30fdb28464cbdd62fca1bc21ddbef6f563920a2c" Nov 01 22:56:29.867320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.867283 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8"] Nov 01 22:56:29.872445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:29.872414 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e202e7-openshift-default-7f86fc79c-4jbd8"] Nov 01 22:56:30.057661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:30.057618 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:30.739902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:30.739871 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" path="/var/lib/kubelet/pods/b153d5e5-f84f-48a3-baea-50a3fbce1b12/volumes" Nov 01 22:56:30.972245 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:30.972216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:31.888535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:31.888500 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:32.658507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:32.658468 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:33.085319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:33.085274 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" podUID="b030a076-acd3-4f9a-b258-770566ec72b6" containerName="payload-processing" containerID="cri-o://11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f" gracePeriod=25 Nov 01 22:56:33.439640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:33.439599 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:33.862909 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:33.862811 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf"] Nov 01 22:56:34.001899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:34.001849 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:34.105937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:34.105904 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:34.310312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:34.310277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:35.265815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:35.265782 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:35.983751 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:35.983709 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:36.909586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:36.909536 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:37.619966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:37.619922 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:38.263794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:38.263759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:38.853903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:38.853861 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:38.863720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:38.863686 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" podUID="44d06625-2be0-4199-b8bb-91fd93ef016a" containerName="payload-processing" containerID="cri-o://e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199" gracePeriod=25 Nov 01 22:56:39.210224 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.210189 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:56:39.288562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.288528 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d06625-2be0-4199-b8bb-91fd93ef016a-kube-api-access-2ghb4\" (UniqueName: \"kubernetes.io/projected/44d06625-2be0-4199-b8bb-91fd93ef016a-kube-api-access-2ghb4\") pod \"44d06625-2be0-4199-b8bb-91fd93ef016a\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " Nov 01 22:56:39.289100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.288569 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d06625-2be0-4199-b8bb-91fd93ef016a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d06625-2be0-4199-b8bb-91fd93ef016a-plugins-config-volume\") pod \"44d06625-2be0-4199-b8bb-91fd93ef016a\" (UID: \"44d06625-2be0-4199-b8bb-91fd93ef016a\") " Nov 01 22:56:39.289100 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.289071 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d06625-2be0-4199-b8bb-91fd93ef016a-plugins-config-volume" pod "44d06625-2be0-4199-b8bb-91fd93ef016a" (UID: "44d06625-2be0-4199-b8bb-91fd93ef016a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:39.289471 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.289452 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d06625-2be0-4199-b8bb-91fd93ef016a-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:39.291183 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.291154 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d06625-2be0-4199-b8bb-91fd93ef016a-kube-api-access-2ghb4" pod "44d06625-2be0-4199-b8bb-91fd93ef016a" (UID: "44d06625-2be0-4199-b8bb-91fd93ef016a"). InnerVolumeSpecName "kube-api-access-2ghb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:39.391805 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.391770 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghb4\" (UniqueName: \"kubernetes.io/projected/44d06625-2be0-4199-b8bb-91fd93ef016a-kube-api-access-2ghb4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:39.556000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.555967 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:39.968696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.968663 4981 generic.go:356] "Generic (PLEG): container finished" podID="44d06625-2be0-4199-b8bb-91fd93ef016a" containerID="e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199" exitCode=0 Nov 01 22:56:39.968917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.968713 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" Nov 01 22:56:39.968917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.968714 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" event={"ID":"44d06625-2be0-4199-b8bb-91fd93ef016a","Type":"ContainerDied","Data":"e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199"} Nov 01 22:56:39.968917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.968785 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf" event={"ID":"44d06625-2be0-4199-b8bb-91fd93ef016a","Type":"ContainerDied","Data":"de163947ee4f051f175bd76eb85fb3101b751fb9be23bafa3c3c4eb4b6ba484d"} Nov 01 22:56:39.968917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.968805 4981 scope.go:122] "RemoveContainer" containerID="e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199" Nov 01 22:56:39.981979 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.981948 4981 scope.go:122] "RemoveContainer" containerID="e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199" Nov 01 22:56:39.982550 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:56:39.982521 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199\": container with ID starting with e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199 not found: ID does not exist" containerID="e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199" Nov 01 22:56:39.982687 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.982558 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199"} err="failed to get container status \"e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199\": rpc error: code = NotFound desc = could not find container \"e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199\": container with ID starting with e78b2d064d34312156b3f9519a38460ac99492179c305101b48beca71c2f6199 not found: ID does not exist" Nov 01 22:56:39.998411 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:39.998376 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf"] Nov 01 22:56:40.003846 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:40.003811 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f07902-6b7dddd6f9-rbjdf"] Nov 01 22:56:40.366336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:40.366248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:40.740794 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:40.740759 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d06625-2be0-4199-b8bb-91fd93ef016a" path="/var/lib/kubelet/pods/44d06625-2be0-4199-b8bb-91fd93ef016a/volumes" Nov 01 22:56:41.145296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:41.145204 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:42.013601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:42.012739 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:42.486671 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:42.486631 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" podUID="b030a076-acd3-4f9a-b258-770566ec72b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:42.764499 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:42.764384 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:43.529674 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:43.529620 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:44.064626 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:44.064563 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:44.204070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:44.204039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:44.324477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:44.324383 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:44.644472 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:44.644377 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s"] Nov 01 22:56:44.649442 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:56:44.649402 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:56:44.655064 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:44.655036 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e716d-dfs7s"] Nov 01 22:56:44.740081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:44.740049 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31acdd55-098c-4a00-850d-6dbe02f7638a" path="/var/lib/kubelet/pods/31acdd55-098c-4a00-850d-6dbe02f7638a/volumes" Nov 01 22:56:44.824071 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:44.824035 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:45.009327 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:45.009297 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:45.790069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:45.790031 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:46.450175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:46.450139 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:47.217342 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:47.217311 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:48.039116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:48.039079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:48.857954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:48.857915 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:49.653969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:49.653928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:49.730735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:49.730663 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:49.844891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:49.844792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:50.545482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:50.545445 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:50.964904 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:50.964865 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:51.064535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:51.064490 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:51.289453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:51.289365 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:51.895292 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:51.895262 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:52.486551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:52.486508 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" podUID="b030a076-acd3-4f9a-b258-770566ec72b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:56:52.573937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:52.573899 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:52.674429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:52.674395 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:53.378718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.378684 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:53.416372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.416343 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:56:53.515681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.515485 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b030a076-acd3-4f9a-b258-770566ec72b6-kube-api-access-6txxm\" (UniqueName: \"kubernetes.io/projected/b030a076-acd3-4f9a-b258-770566ec72b6-kube-api-access-6txxm\") pod \"b030a076-acd3-4f9a-b258-770566ec72b6\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " Nov 01 22:56:53.515681 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.515614 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b030a076-acd3-4f9a-b258-770566ec72b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b030a076-acd3-4f9a-b258-770566ec72b6-plugins-config-volume\") pod \"b030a076-acd3-4f9a-b258-770566ec72b6\" (UID: \"b030a076-acd3-4f9a-b258-770566ec72b6\") " Nov 01 22:56:53.517614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.517539 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b030a076-acd3-4f9a-b258-770566ec72b6-plugins-config-volume" pod "b030a076-acd3-4f9a-b258-770566ec72b6" (UID: "b030a076-acd3-4f9a-b258-770566ec72b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:56:53.523425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.523341 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b030a076-acd3-4f9a-b258-770566ec72b6-kube-api-access-6txxm" pod "b030a076-acd3-4f9a-b258-770566ec72b6" (UID: "b030a076-acd3-4f9a-b258-770566ec72b6"). InnerVolumeSpecName "kube-api-access-6txxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:56:53.618526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.618451 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6txxm\" (UniqueName: \"kubernetes.io/projected/b030a076-acd3-4f9a-b258-770566ec72b6-kube-api-access-6txxm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:53.618526 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:53.618482 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b030a076-acd3-4f9a-b258-770566ec72b6-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:56:54.001768 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.001734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:54.110212 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.110174 4981 generic.go:356] "Generic (PLEG): container finished" podID="b030a076-acd3-4f9a-b258-770566ec72b6" containerID="11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f" exitCode=0 Nov 01 22:56:54.110419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.110232 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" Nov 01 22:56:54.110419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.110263 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" event={"ID":"b030a076-acd3-4f9a-b258-770566ec72b6","Type":"ContainerDied","Data":"11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f"} Nov 01 22:56:54.110419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.110338 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f" event={"ID":"b030a076-acd3-4f9a-b258-770566ec72b6","Type":"ContainerDied","Data":"90d17d8d5be003b748f2b763b4a999d773ad78342f268986780df65ad021a5f7"} Nov 01 22:56:54.110419 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.110369 4981 scope.go:122] "RemoveContainer" containerID="11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f" Nov 01 22:56:54.129145 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.129028 4981 scope.go:122] "RemoveContainer" containerID="11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f" Nov 01 22:56:54.129509 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:56:54.129439 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f\": container with ID starting with 11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f not found: ID does not exist" containerID="11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f" Nov 01 22:56:54.129509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.129474 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f"} err="failed to get container status \"11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f\": rpc error: code = NotFound desc = could not find container \"11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f\": container with ID starting with 11dbd38f850d80e8afc2833a43088df6be05fe30aeceb6b2734968b275ac2e6f not found: ID does not exist" Nov 01 22:56:54.141296 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.141220 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f"] Nov 01 22:56:54.160610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.157929 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0e716d-64677df9b9-kcx8f"] Nov 01 22:56:54.179602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.168801 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:54.291656 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.285509 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:54.581775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.581657 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt"] Nov 01 22:56:54.619654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.617773 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f07902-gfppt"] Nov 01 22:56:54.656836 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.656806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:54.792072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.789448 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd42122-3520-40fd-a632-6b28769b6730" path="/var/lib/kubelet/pods/0dd42122-3520-40fd-a632-6b28769b6730/volumes" Nov 01 22:56:54.792072 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:54.790002 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b030a076-acd3-4f9a-b258-770566ec72b6" path="/var/lib/kubelet/pods/b030a076-acd3-4f9a-b258-770566ec72b6/volumes" Nov 01 22:56:55.001469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:55.001437 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:55.218257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:55.215052 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz"] Nov 01 22:56:55.218257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:55.215315 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" containerID="cri-o://1afc26661d7f405df2f5cfacdf5416d5432e14c66f6ba350aef96d3f412eb864" gracePeriod=30 Nov 01 22:56:55.382295 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:55.382212 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:55.481302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:55.481266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:55.581041 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:55.581007 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:56.014010 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:56.013975 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg"] Nov 01 22:56:56.014468 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:56.014274 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" containerID="cri-o://468e13ea71f93d4c825adff3df1cc942e10b8ef9bb4bf9a849ed7d8459552eac" gracePeriod=30 Nov 01 22:56:56.468128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:56.468096 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:57.215314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:57.215284 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:57.984750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:57.984718 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:58.736009 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:58.735970 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:56:59.566724 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:56:59.566690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:00.371680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.369269 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:00.586242 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.586211 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:57:00.604851 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.604820 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-socket\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605029 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.604870 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-data\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605095 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605030 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-token\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-token\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605128 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-socket" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:00.605172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605162 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-podinfo\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605170 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-data" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:00.605258 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605217 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istiod-ca-cert\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605263 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-certs\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605369 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-credential-socket\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605445 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605409 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-kube-api-access-wfkvr\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-kube-api-access-wfkvr\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605525 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-envoy\") pod \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\" (UID: \"ee8d8482-22c2-4804-a7db-fe1dbcf168d7\") " Nov 01 22:57:00.605543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605531 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-certs" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:00.605716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605597 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istiod-ca-cert" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:57:00.605783 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.605731 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-credential-socket" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:00.606283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.606260 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.606283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.606280 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.606491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.606289 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.606491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.606299 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.606491 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.606310 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.607385 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.607361 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-token" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:00.607667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.607642 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-podinfo" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:57:00.608026 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.608008 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-envoy" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:00.608110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.608027 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-kube-api-access-wfkvr" pod "ee8d8482-22c2-4804-a7db-fe1dbcf168d7" (UID: "ee8d8482-22c2-4804-a7db-fe1dbcf168d7"). InnerVolumeSpecName "kube-api-access-wfkvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:00.708555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.708521 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfkvr\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-kube-api-access-wfkvr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.708555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.708551 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.708555 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.708561 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:00.708800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:00.708586 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8d8482-22c2-4804-a7db-fe1dbcf168d7-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.054402 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.054371 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:01.152639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.152072 4981 scope.go:122] "RemoveContainer" containerID="1afc26661d7f405df2f5cfacdf5416d5432e14c66f6ba350aef96d3f412eb864" Nov 01 22:57:01.195708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.195400 4981 generic.go:356] "Generic (PLEG): container finished" podID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerID="468e13ea71f93d4c825adff3df1cc942e10b8ef9bb4bf9a849ed7d8459552eac" exitCode=0 Nov 01 22:57:01.195708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.195452 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" event={"ID":"ede5d133-8820-48a7-8b9b-6c6b4fef23ff","Type":"ContainerDied","Data":"468e13ea71f93d4c825adff3df1cc942e10b8ef9bb4bf9a849ed7d8459552eac"} Nov 01 22:57:01.197048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.197012 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" event={"ID":"ee8d8482-22c2-4804-a7db-fe1dbcf168d7","Type":"ContainerDied","Data":"1afc26661d7f405df2f5cfacdf5416d5432e14c66f6ba350aef96d3f412eb864"} Nov 01 22:57:01.197179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.197077 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" Nov 01 22:57:01.197179 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.197082 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz" event={"ID":"ee8d8482-22c2-4804-a7db-fe1dbcf168d7","Type":"ContainerDied","Data":"edb71e9c532a8016c3253559d0775ad02744bba9d751de1ebe57bc5c019230c6"} Nov 01 22:57:01.228057 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.228030 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz"] Nov 01 22:57:01.234171 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.234144 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-jq4jz"] Nov 01 22:57:01.384786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.384758 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:57:01.425892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.425853 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-certs\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.425976 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-podinfo\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426049 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-kube-api-access-9hfgw\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-kube-api-access-9hfgw\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426189 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-data\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426192 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-certs" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.426347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426282 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-envoy\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426338 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-credential-socket\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426595 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-token\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-token\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426641 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426612 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-data" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.426829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426709 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istiod-ca-cert\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.426829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426750 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-credential-socket" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.426829 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.426797 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-socket\") pod \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\" (UID: \"ede5d133-8820-48a7-8b9b-6c6b4fef23ff\") " Nov 01 22:57:01.428209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.428045 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-socket" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.428209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.428104 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.428209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.428129 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.428209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.428145 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.428431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.428229 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istiod-ca-cert" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:57:01.429162 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.429140 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-podinfo" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:57:01.429673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.429647 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-kube-api-access-9hfgw" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "kube-api-access-9hfgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:01.429769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.429687 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-token" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:01.429928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.429909 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-envoy" pod "ede5d133-8820-48a7-8b9b-6c6b4fef23ff" (UID: "ede5d133-8820-48a7-8b9b-6c6b4fef23ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:01.530181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.530116 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.530181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.530160 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.530181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.530176 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.530504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.530189 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hfgw\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-kube-api-access-9hfgw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.530504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.530207 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.530504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.530220 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede5d133-8820-48a7-8b9b-6c6b4fef23ff-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:01.759538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.759504 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:01.864597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:01.864548 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:02.045503 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.045403 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx"] Nov 01 22:57:02.045746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.045711 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" containerID="cri-o://648f23c0a664b56e7839a63dea18b6f9272ba68a35608308128d861a0b57d205" gracePeriod=30 Nov 01 22:57:02.083423 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.083385 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:02.307502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.306791 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" event={"ID":"ede5d133-8820-48a7-8b9b-6c6b4fef23ff","Type":"ContainerDied","Data":"d900504021a407fc17e2ad1e4ad907680fda52e6f1bbe777739f774dc3c25b46"} Nov 01 22:57:02.307502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.306872 4981 scope.go:122] "RemoveContainer" containerID="468e13ea71f93d4c825adff3df1cc942e10b8ef9bb4bf9a849ed7d8459552eac" Nov 01 22:57:02.307502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.307098 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg" Nov 01 22:57:02.686350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.686272 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:02.709692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.709656 4981 patch_prober.go:29] interesting pod/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:57:02.709879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.709715 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:57:02.786380 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.786344 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" path="/var/lib/kubelet/pods/ee8d8482-22c2-4804-a7db-fe1dbcf168d7/volumes" Nov 01 22:57:02.852281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:02.852246 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:03.307567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:03.307531 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:03.410948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:03.410819 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:03.861204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:03.861172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:04.009326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:04.009272 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg"] Nov 01 22:57:04.013767 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:04.013740 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-9c7wg"] Nov 01 22:57:04.649684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:04.649630 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:04.786279 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:04.786245 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" path="/var/lib/kubelet/pods/ede5d133-8820-48a7-8b9b-6c6b4fef23ff/volumes" Nov 01 22:57:05.423253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:05.423200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:06.058864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:06.058834 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:06.861968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:06.861934 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:07.365128 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.365098 4981 generic.go:356] "Generic (PLEG): container finished" podID="2d275241-c066-4ab8-828a-b6acab847630" containerID="648f23c0a664b56e7839a63dea18b6f9272ba68a35608308128d861a0b57d205" exitCode=0 Nov 01 22:57:07.365297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.365141 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" event={"ID":"2d275241-c066-4ab8-828a-b6acab847630","Type":"ContainerDied","Data":"648f23c0a664b56e7839a63dea18b6f9272ba68a35608308128d861a0b57d205"} Nov 01 22:57:07.365297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.365183 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" event={"ID":"2d275241-c066-4ab8-828a-b6acab847630","Type":"ContainerDied","Data":"3fbe34e69ffd2c511f26ec3ba98b875344033094d0be40421cd0115bfa941d28"} Nov 01 22:57:07.365297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.365199 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fbe34e69ffd2c511f26ec3ba98b875344033094d0be40421cd0115bfa941d28" Nov 01 22:57:07.365297 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.365211 4981 scope.go:122] "RemoveContainer" containerID="b4b94f042fb1eca9188b89ba18bf6bd6cba017a9ba67d22d51dc06c84300f8db" Nov 01 22:57:07.373155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.373133 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:57:07.534024 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.533990 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d275241-c066-4ab8-828a-b6acab847630-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d275241-c066-4ab8-828a-b6acab847630-istio-podinfo\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534069 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-data\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534182 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-istio-token\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-istio-token\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534290 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-kube-api-access-wbfv6\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-kube-api-access-wbfv6\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534362 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-data" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:07.534498 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534404 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-envoy\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534496 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d275241-c066-4ab8-828a-b6acab847630-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d275241-c066-4ab8-828a-b6acab847630-istiod-ca-cert\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534696 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534651 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-credential-socket\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534778 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-socket\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.534947 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534890 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-certs\") pod \"2d275241-c066-4ab8-828a-b6acab847630\" (UID: \"2d275241-c066-4ab8-828a-b6acab847630\") " Nov 01 22:57:07.535004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.534949 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d275241-c066-4ab8-828a-b6acab847630-istiod-ca-cert" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:57:07.535086 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535050 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-socket" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:07.535159 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535124 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-credential-socket" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:07.535267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535249 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-certs" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:07.535725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535704 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d275241-c066-4ab8-828a-b6acab847630-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.535818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535727 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.535818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535743 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.535818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535756 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.535818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.535768 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.536451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.536421 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d275241-c066-4ab8-828a-b6acab847630-istio-podinfo" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:57:07.536553 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.536510 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-istio-token" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:07.536857 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.536838 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-kube-api-access-wbfv6" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "kube-api-access-wbfv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:07.537016 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.537000 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-envoy" pod "2d275241-c066-4ab8-828a-b6acab847630" (UID: "2d275241-c066-4ab8-828a-b6acab847630"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:57:07.637777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.637743 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.637777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.637774 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbfv6\" (UniqueName: \"kubernetes.io/projected/2d275241-c066-4ab8-828a-b6acab847630-kube-api-access-wbfv6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.637777 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.637784 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d275241-c066-4ab8-828a-b6acab847630-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.638040 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.637793 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d275241-c066-4ab8-828a-b6acab847630-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:07.685731 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.685698 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:07.847652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:07.845508 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:08.374661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:08.374634 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx" Nov 01 22:57:08.492588 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:08.492552 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx"] Nov 01 22:57:08.502520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:08.502490 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f07902-openshift-default-d865f8dbf-xnkhx"] Nov 01 22:57:08.723952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:08.723915 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:08.746328 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:08.746289 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d275241-c066-4ab8-828a-b6acab847630" path="/var/lib/kubelet/pods/2d275241-c066-4ab8-828a-b6acab847630/volumes" Nov 01 22:57:08.958680 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:08.958645 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:09.058387 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.058305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:09.059088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.059057 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6"] Nov 01 22:57:09.060673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.060647 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" Nov 01 22:57:09.060839 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.060822 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" Nov 01 22:57:09.061004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.060983 4981 cpu_manager.go:414] "containerMap: removing container" podUID="44d06625-2be0-4199-b8bb-91fd93ef016a" containerName="payload-processing" Nov 01 22:57:09.061004 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061001 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d06625-2be0-4199-b8bb-91fd93ef016a" containerName="payload-processing" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061026 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0dd42122-3520-40fd-a632-6b28769b6730" containerName="revoke-keys" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061034 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd42122-3520-40fd-a632-6b28769b6730" containerName="revoke-keys" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061059 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b030a076-acd3-4f9a-b258-770566ec72b6" containerName="payload-processing" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061067 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b030a076-acd3-4f9a-b258-770566ec72b6" containerName="payload-processing" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061085 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061092 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061131 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061139 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" Nov 01 22:57:09.061168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061165 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061173 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061208 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061216 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061441 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061460 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061477 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d275241-c066-4ab8-828a-b6acab847630" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061493 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d06625-2be0-4199-b8bb-91fd93ef016a" containerName="payload-processing" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061506 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd42122-3520-40fd-a632-6b28769b6730" containerName="revoke-keys" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061518 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b030a076-acd3-4f9a-b258-770566ec72b6" containerName="payload-processing" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061533 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede5d133-8820-48a7-8b9b-6c6b4fef23ff" containerName="istio-proxy" Nov 01 22:57:09.061667 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.061550 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8d8482-22c2-4804-a7db-fe1dbcf168d7" containerName="istio-proxy" Nov 01 22:57:09.069705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.069679 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.082116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.082086 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6"] Nov 01 22:57:09.130966 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.130921 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-788d7b-dockercfg-s77jw\"" Nov 01 22:57:09.131186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.131037 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-788d7b\"" Nov 01 22:57:09.207603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.206972 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcg5q\" (UniqueName: \"kubernetes.io/projected/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-kube-api-access-bcg5q\") pod \"payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.207603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.207242 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.310720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.308944 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcg5q\" (UniqueName: \"kubernetes.io/projected/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-kube-api-access-bcg5q\") pod \"payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.310720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.309244 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.310720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.310345 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.344596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.344512 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcg5q\" (UniqueName: \"kubernetes.io/projected/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-kube-api-access-bcg5q\") pod \"payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.469689 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.468227 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:09.740121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.740092 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:09.786827 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:57:09.786797 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5f7521_0a94_4d62_b9d5_772c9d1f62dd.slice/crio-9620bb11e20df85072d584e10874ac127d70538a93899b373fda03134b9296e4 WatchSource:0}: Error finding container 9620bb11e20df85072d584e10874ac127d70538a93899b373fda03134b9296e4: Status 404 returned error can't find the container with id 9620bb11e20df85072d584e10874ac127d70538a93899b373fda03134b9296e4 Nov 01 22:57:09.787151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:09.787127 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6"] Nov 01 22:57:10.431929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:10.431889 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" event={"ID":"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd","Type":"ContainerStarted","Data":"16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b"} Nov 01 22:57:10.432123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:10.431949 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" event={"ID":"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd","Type":"ContainerStarted","Data":"9620bb11e20df85072d584e10874ac127d70538a93899b373fda03134b9296e4"} Nov 01 22:57:10.432123 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:10.431970 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:10.454021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:10.453963 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" podStartSLOduration=1.4539429400000001 podStartE2EDuration="1.45394294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:10.453328092 +0000 UTC m=+1995.881206023" watchObservedRunningTime="2026-11-01 22:57:10.45394294 +0000 UTC m=+1995.881820871" Nov 01 22:57:10.482378 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:10.482334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:11.272345 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:11.272282 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:11.873281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:11.873251 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:12.830036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:12.830005 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:13.265558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:13.265526 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:13.776859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:13.776824 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:13.884482 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:13.884454 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:14.528616 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:57:14.528539 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:57:14.642601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:14.633252 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:15.156620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:15.150056 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:16.074617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:16.073766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:16.927879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:16.927845 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:17.771226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:17.771171 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:18.549600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:18.544915 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:19.802824 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:19.802752 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:19.968828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:19.968788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:20.668254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:20.668217 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:21.364017 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:21.363982 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:21.479609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:21.478857 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 22:57:22.148175 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:22.148144 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:22.750874 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:22.750834 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:23.429451 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:23.429419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:24.379729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:24.379693 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:25.049155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.049117 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:25.675566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.675533 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:25.676546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.676521 4981 generic.go:356] "Generic (PLEG): container finished" podID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerID="c4db3e1058867c2a345a8173ebbe5ab99760039ae8f872ec0b581e113bb0073f" exitCode=0 Nov 01 22:57:25.676704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.676620 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" event={"ID":"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b","Type":"ContainerDied","Data":"c4db3e1058867c2a345a8173ebbe5ab99760039ae8f872ec0b581e113bb0073f"} Nov 01 22:57:25.677140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.677125 4981 scope.go:122] "RemoveContainer" containerID="c4db3e1058867c2a345a8173ebbe5ab99760039ae8f872ec0b581e113bb0073f" Nov 01 22:57:25.679157 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.679133 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:57:25.679226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.679198 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:57:25.679267 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:25.679231 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:57:26.360622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:26.360589 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:26.687796 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:26.687759 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" event={"ID":"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b","Type":"ContainerStarted","Data":"69ba2e04743cdf97c483a88633c92dc9424b07e253415785713e3ba7431486c1"} Nov 01 22:57:26.844268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:26.844225 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:57:26.845568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:26.845535 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:57:26.845747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:26.845641 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Nov 01 22:57:27.049132 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:27.049049 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:27.843842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:27.843806 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:57:27.844262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:27.843872 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Nov 01 22:57:27.859882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:27.859858 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:27.975620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:27.975193 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:28.073359 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:28.073328 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:28.739964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:28.739931 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:28.860762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:28.860057 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:57:28.860762 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:28.860128 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Nov 01 22:57:29.019440 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:29.019351 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:29.440599 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:29.429829 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:29.535248 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:29.535220 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:29.695055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:29.694969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:29.846717 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:29.846685 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:57:29.846964 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:29.846942 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Nov 01 22:57:30.551540 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:30.551508 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:30.844185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:30.844104 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:57:30.844185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:30.844169 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Nov 01 22:57:31.255336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:31.255302 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:31.273102 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:31.273077 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:57:31.843632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:31.843561 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Nov 01 22:57:31.844177 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:31.843647 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Nov 01 22:57:31.847775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:31.847749 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:32.574199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:32.574163 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:32.849084 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:32.849006 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:57:32.849929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:32.849908 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 22:57:33.162675 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.162596 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz"] Nov 01 22:57:33.163516 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.163498 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" Nov 01 22:57:33.163566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.163518 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" Nov 01 22:57:33.163712 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.163702 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b153d5e5-f84f-48a3-baea-50a3fbce1b12" containerName="istio-proxy" Nov 01 22:57:33.168186 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.168167 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.174653 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.174627 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:57:33.174823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.174654 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:57:33.177739 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.177715 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz"] Nov 01 22:57:33.253002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.252969 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.253205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.253173 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.253528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.253502 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg4m7\" (UniqueName: \"kubernetes.io/projected/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-kube-api-access-pg4m7\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.355373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.355336 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg4m7\" (UniqueName: \"kubernetes.io/projected/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-kube-api-access-pg4m7\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.355560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.355445 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.355560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.355521 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.356185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.356161 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.357943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.357919 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.367514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.367488 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg4m7\" (UniqueName: \"kubernetes.io/projected/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-kube-api-access-pg4m7\") pod \"maas-api-revoke-sub-1106dd29-cwdvz\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.450532 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.450505 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:33.497856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.497806 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:33.819617 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:57:33.819556 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cc3a9c_1a8a_49ab_b7b6_a5f8ba0d6b6c.slice/crio-5b9044d175708337637c34c445a54af77d624bed903006db340522e48de4511c WatchSource:0}: Error finding container 5b9044d175708337637c34c445a54af77d624bed903006db340522e48de4511c: Status 404 returned error can't find the container with id 5b9044d175708337637c34c445a54af77d624bed903006db340522e48de4511c Nov 01 22:57:33.820364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:33.820336 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz"] Nov 01 22:57:34.160000 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:34.159948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:34.770770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:34.770722 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" event={"ID":"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c","Type":"ContainerStarted","Data":"137a521bc11c85618f451b47966be3a534c2cc5fb2b797d8494ca8302ef3a108"} Nov 01 22:57:34.770968 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:34.770787 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" event={"ID":"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c","Type":"ContainerStarted","Data":"5b9044d175708337637c34c445a54af77d624bed903006db340522e48de4511c"} Nov 01 22:57:34.777151 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:34.777126 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:34.796938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:34.796896 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" podStartSLOduration=1.796882874 podStartE2EDuration="1.796882874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:34.791864411 +0000 UTC m=+2020.219742346" watchObservedRunningTime="2026-11-01 22:57:34.796882874 +0000 UTC m=+2020.224760805" Nov 01 22:57:35.477720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:35.477689 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:35.782615 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:35.782515 4981 generic.go:356] "Generic (PLEG): container finished" podID="92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" containerID="137a521bc11c85618f451b47966be3a534c2cc5fb2b797d8494ca8302ef3a108" exitCode=0 Nov 01 22:57:35.782782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:35.782640 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" event={"ID":"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c","Type":"ContainerDied","Data":"137a521bc11c85618f451b47966be3a534c2cc5fb2b797d8494ca8302ef3a108"} Nov 01 22:57:36.250247 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:36.250216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:36.979431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:36.979407 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:36.980609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:36.980567 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:37.147195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.147111 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-xks-ca\") pod \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " Nov 01 22:57:37.147195 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.147149 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-openshift-ca\") pod \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " Nov 01 22:57:37.147407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.147231 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-kube-api-access-pg4m7\" (UniqueName: \"kubernetes.io/projected/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-kube-api-access-pg4m7\") pod \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\" (UID: \"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c\") " Nov 01 22:57:37.147627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.147595 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-openshift-ca" pod "92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" (UID: "92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:57:37.148350 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.148329 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:37.149252 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.149225 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-xks-ca" pod "92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" (UID: "92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:57:37.149365 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.149345 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-kube-api-access-pg4m7" pod "92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" (UID: "92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c"). InnerVolumeSpecName "kube-api-access-pg4m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:37.250513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.250477 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:37.250513 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.250510 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg4m7\" (UniqueName: \"kubernetes.io/projected/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c-kube-api-access-pg4m7\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:37.653246 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.653213 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:37.802985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.802954 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" Nov 01 22:57:37.802985 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.802967 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz" event={"ID":"92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c","Type":"ContainerDied","Data":"5b9044d175708337637c34c445a54af77d624bed903006db340522e48de4511c"} Nov 01 22:57:37.803222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:37.803011 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b9044d175708337637c34c445a54af77d624bed903006db340522e48de4511c" Nov 01 22:57:38.452188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:38.452146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:39.041958 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:39.041923 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:39.812708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:39.812675 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:40.548764 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:40.548730 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:41.172684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:41.172647 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:41.954518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:41.954480 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:42.768288 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:42.768250 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:43.513254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:43.513196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:43.955011 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:43.954981 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:44.430470 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.430430 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5"] Nov 01 22:57:44.432614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.432568 4981 cpu_manager.go:414] "containerMap: removing container" podUID="92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" containerName="revoke-keys" Nov 01 22:57:44.432614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.432614 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" containerName="revoke-keys" Nov 01 22:57:44.433083 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.433062 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" containerName="revoke-keys" Nov 01 22:57:44.443518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.443493 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.448074 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.448042 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5"] Nov 01 22:57:44.461027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.456952 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:57:44.461027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.457070 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:57:44.507485 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.507436 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.507698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.507513 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.508069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.508038 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jww\" (UniqueName: \"kubernetes.io/projected/2a6185c2-0df8-46e9-aabc-43f55718935e-kube-api-access-t4jww\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.530970 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:57:44.530938 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:57:44.562139 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.562113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:44.623595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.619033 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.623595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.619121 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.623595 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.619252 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jww\" (UniqueName: \"kubernetes.io/projected/2a6185c2-0df8-46e9-aabc-43f55718935e-kube-api-access-t4jww\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.631003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.630958 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.640798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.640674 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.642640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.642591 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jww\" (UniqueName: \"kubernetes.io/projected/2a6185c2-0df8-46e9-aabc-43f55718935e-kube-api-access-t4jww\") pod \"maas-api-revoke-sub-be8b244a-vtkl5\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:44.765109 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:44.765037 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:45.053708 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:57:45.053679 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6185c2_0df8_46e9_aabc_43f55718935e.slice/crio-b93b34672a5a4fa22409e8eb6706745ad396fbfd888046500332e5346c74054a WatchSource:0}: Error finding container b93b34672a5a4fa22409e8eb6706745ad396fbfd888046500332e5346c74054a: Status 404 returned error can't find the container with id b93b34672a5a4fa22409e8eb6706745ad396fbfd888046500332e5346c74054a Nov 01 22:57:45.055614 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:45.055570 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5"] Nov 01 22:57:45.595013 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:45.594976 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:45.878984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:45.878897 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" event={"ID":"2a6185c2-0df8-46e9-aabc-43f55718935e","Type":"ContainerStarted","Data":"632841cef7982f776c32e5565a9ea708aec606e13f42d21e4c74468304c85fcb"} Nov 01 22:57:45.878984 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:45.878961 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" event={"ID":"2a6185c2-0df8-46e9-aabc-43f55718935e","Type":"ContainerStarted","Data":"b93b34672a5a4fa22409e8eb6706745ad396fbfd888046500332e5346c74054a"} Nov 01 22:57:45.934917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:45.934865 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" podStartSLOduration=1.934852011 podStartE2EDuration="1.934852011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:45.928231758 +0000 UTC m=+2031.356109691" watchObservedRunningTime="2026-11-01 22:57:45.934852011 +0000 UTC m=+2031.362729942" Nov 01 22:57:46.417375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:46.417345 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:46.889898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:46.889864 4981 generic.go:356] "Generic (PLEG): container finished" podID="2a6185c2-0df8-46e9-aabc-43f55718935e" containerID="632841cef7982f776c32e5565a9ea708aec606e13f42d21e4c74468304c85fcb" exitCode=0 Nov 01 22:57:46.890085 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:46.889906 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" event={"ID":"2a6185c2-0df8-46e9-aabc-43f55718935e","Type":"ContainerDied","Data":"632841cef7982f776c32e5565a9ea708aec606e13f42d21e4c74468304c85fcb"} Nov 01 22:57:47.157987 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:47.157902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:47.958316 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:47.958274 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:48.137637 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.137611 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:48.294477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.294441 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-xks-ca\") pod \"2a6185c2-0df8-46e9-aabc-43f55718935e\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " Nov 01 22:57:48.294692 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.294569 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6185c2-0df8-46e9-aabc-43f55718935e-kube-api-access-t4jww\" (UniqueName: \"kubernetes.io/projected/2a6185c2-0df8-46e9-aabc-43f55718935e-kube-api-access-t4jww\") pod \"2a6185c2-0df8-46e9-aabc-43f55718935e\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " Nov 01 22:57:48.294812 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.294788 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-openshift-ca\") pod \"2a6185c2-0df8-46e9-aabc-43f55718935e\" (UID: \"2a6185c2-0df8-46e9-aabc-43f55718935e\") " Nov 01 22:57:48.295122 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.295094 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-openshift-ca" pod "2a6185c2-0df8-46e9-aabc-43f55718935e" (UID: "2a6185c2-0df8-46e9-aabc-43f55718935e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:57:48.295606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.295570 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:48.296863 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.296827 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6185c2-0df8-46e9-aabc-43f55718935e-kube-api-access-t4jww" pod "2a6185c2-0df8-46e9-aabc-43f55718935e" (UID: "2a6185c2-0df8-46e9-aabc-43f55718935e"). InnerVolumeSpecName "kube-api-access-t4jww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:57:48.297104 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.297079 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-xks-ca" pod "2a6185c2-0df8-46e9-aabc-43f55718935e" (UID: "2a6185c2-0df8-46e9-aabc-43f55718935e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:57:48.397862 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.397828 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a6185c2-0df8-46e9-aabc-43f55718935e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:48.397862 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.397857 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jww\" (UniqueName: \"kubernetes.io/projected/2a6185c2-0df8-46e9-aabc-43f55718935e-kube-api-access-t4jww\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:57:48.653727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.653649 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:48.909941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.909858 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" Nov 01 22:57:48.909941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.909887 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5" event={"ID":"2a6185c2-0df8-46e9-aabc-43f55718935e","Type":"ContainerDied","Data":"b93b34672a5a4fa22409e8eb6706745ad396fbfd888046500332e5346c74054a"} Nov 01 22:57:48.909941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:48.909937 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93b34672a5a4fa22409e8eb6706745ad396fbfd888046500332e5346c74054a" Nov 01 22:57:49.545598 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:49.544895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:50.387291 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:50.387259 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:51.081949 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:51.081913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:51.777320 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:51.777293 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:52.531734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:52.531699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:53.322324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:53.322266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:53.422520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:53.422477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:53.660371 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:53.660298 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:54.081967 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:54.081929 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:54.178729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:54.178701 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:54.288084 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:54.288049 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:55.155205 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:55.155169 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:55.856373 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:55.856330 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:56.727275 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:56.727233 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:57.637225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:57.637196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:58.269708 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:58.269672 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:58.364825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:58.364791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:58.472821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:58.472790 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:59.347753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.347718 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:59.446930 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.446897 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:59.892823 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.892792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:57:59.976270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.976238 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf"] Nov 01 22:57:59.977178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.977157 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2a6185c2-0df8-46e9-aabc-43f55718935e" containerName="revoke-keys" Nov 01 22:57:59.977178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.977176 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6185c2-0df8-46e9-aabc-43f55718935e" containerName="revoke-keys" Nov 01 22:57:59.977367 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.977357 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6185c2-0df8-46e9-aabc-43f55718935e" containerName="revoke-keys" Nov 01 22:57:59.982172 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.982150 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:57:59.993936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.993906 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d73b25\"" Nov 01 22:57:59.994115 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.993932 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d73b25-dockercfg-hm5m7\"" Nov 01 22:57:59.994202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:57:59.994125 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf"] Nov 01 22:58:00.027160 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.027120 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:00.086769 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.086719 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adbe704-d36b-4285-881d-032098ab3cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:00.087079 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.087054 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4cd6\" (UniqueName: \"kubernetes.io/projected/6adbe704-d36b-4285-881d-032098ab3cff-kube-api-access-w4cd6\") pod \"payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:00.255601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.200241 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4cd6\" (UniqueName: \"kubernetes.io/projected/6adbe704-d36b-4285-881d-032098ab3cff-kube-api-access-w4cd6\") pod \"payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:00.277596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.273799 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adbe704-d36b-4285-881d-032098ab3cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:00.300603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.274914 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adbe704-d36b-4285-881d-032098ab3cff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:00.399605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.399304 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4cd6\" (UniqueName: \"kubernetes.io/projected/6adbe704-d36b-4285-881d-032098ab3cff-kube-api-access-w4cd6\") pod \"payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:00.599941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.599852 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:00.682241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.682161 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:00.894263 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:58:00.894223 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adbe704_d36b_4285_881d_032098ab3cff.slice/crio-b24a7243be4dc3b30344666fda5260b5d641f6e9068d118fd919ba123de85745 WatchSource:0}: Error finding container b24a7243be4dc3b30344666fda5260b5d641f6e9068d118fd919ba123de85745: Status 404 returned error can't find the container with id b24a7243be4dc3b30344666fda5260b5d641f6e9068d118fd919ba123de85745 Nov 01 22:58:00.896280 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:00.896248 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf"] Nov 01 22:58:01.031270 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.031228 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" event={"ID":"6adbe704-d36b-4285-881d-032098ab3cff","Type":"ContainerStarted","Data":"4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea"} Nov 01 22:58:01.031444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.031295 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" event={"ID":"6adbe704-d36b-4285-881d-032098ab3cff","Type":"ContainerStarted","Data":"b24a7243be4dc3b30344666fda5260b5d641f6e9068d118fd919ba123de85745"} Nov 01 22:58:01.031444 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.031325 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:01.054650 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.054591 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" podStartSLOduration=2.054559364 podStartE2EDuration="2.054559364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:01.048021038 +0000 UTC m=+2046.475898970" watchObservedRunningTime="2026-11-01 22:58:01.054559364 +0000 UTC m=+2046.482437296" Nov 01 22:58:01.242469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.242437 4981 scope.go:122] "RemoveContainer" containerID="648f23c0a664b56e7839a63dea18b6f9272ba68a35608308128d861a0b57d205" Nov 01 22:58:01.402628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.402560 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:01.750299 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.750253 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt"] Nov 01 22:58:01.759638 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.757963 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.762899 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.762847 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:58:01.763407 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.763369 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:58:01.764509 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.764483 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt"] Nov 01 22:58:01.811431 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.811396 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:01.838185 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.838149 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jkr\" (UniqueName: \"kubernetes.io/projected/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-kube-api-access-64jkr\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.838394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.838371 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.838507 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.838492 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.940941 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.940892 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.941161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.941013 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.941349 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.941325 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jkr\" (UniqueName: \"kubernetes.io/projected/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-kube-api-access-64jkr\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.942281 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.942254 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.946310 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.944790 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:01.950568 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:01.950540 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jkr\" (UniqueName: \"kubernetes.io/projected/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-kube-api-access-64jkr\") pod \"maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:02.077103 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:02.077019 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:02.374090 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:58:02.374061 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1de9e2_97ac_4a1c_9690_cfc619c9eca2.slice/crio-fb57aef44eed654753e4133e283dd7bb6f01bf56956cd3b2c8d2864513a594db WatchSource:0}: Error finding container fb57aef44eed654753e4133e283dd7bb6f01bf56956cd3b2c8d2864513a594db: Status 404 returned error can't find the container with id fb57aef44eed654753e4133e283dd7bb6f01bf56956cd3b2c8d2864513a594db Nov 01 22:58:02.374848 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:02.374825 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt"] Nov 01 22:58:02.689433 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:02.689399 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:03.051736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:03.051639 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" event={"ID":"de1de9e2-97ac-4a1c-9690-cfc619c9eca2","Type":"ContainerStarted","Data":"0f7a60c70f9c1c6f10ab89c4c6b33aea020365aa3906875a6f143f239442033e"} Nov 01 22:58:03.051736 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:03.051703 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" event={"ID":"de1de9e2-97ac-4a1c-9690-cfc619c9eca2","Type":"ContainerStarted","Data":"fb57aef44eed654753e4133e283dd7bb6f01bf56956cd3b2c8d2864513a594db"} Nov 01 22:58:03.073386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:03.073338 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" podStartSLOduration=2.073325027 podStartE2EDuration="2.073325027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:03.068692525 +0000 UTC m=+2048.496570456" watchObservedRunningTime="2026-11-01 22:58:03.073325027 +0000 UTC m=+2048.501202957" Nov 01 22:58:03.396240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:03.396205 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:04.065030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:04.065000 4981 generic.go:356] "Generic (PLEG): container finished" podID="de1de9e2-97ac-4a1c-9690-cfc619c9eca2" containerID="0f7a60c70f9c1c6f10ab89c4c6b33aea020365aa3906875a6f143f239442033e" exitCode=0 Nov 01 22:58:04.065432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:04.065063 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" event={"ID":"de1de9e2-97ac-4a1c-9690-cfc619c9eca2","Type":"ContainerDied","Data":"0f7a60c70f9c1c6f10ab89c4c6b33aea020365aa3906875a6f143f239442033e"} Nov 01 22:58:04.162337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:04.162303 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:04.876962 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:04.876928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:05.285396 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.285366 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:05.421188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.421100 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-openshift-ca\") pod \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " Nov 01 22:58:05.421448 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.421421 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-openshift-ca" pod "de1de9e2-97ac-4a1c-9690-cfc619c9eca2" (UID: "de1de9e2-97ac-4a1c-9690-cfc619c9eca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:05.421528 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.421488 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-xks-ca\") pod \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " Nov 01 22:58:05.421621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.421589 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-kube-api-access-64jkr\" (UniqueName: \"kubernetes.io/projected/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-kube-api-access-64jkr\") pod \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\" (UID: \"de1de9e2-97ac-4a1c-9690-cfc619c9eca2\") " Nov 01 22:58:05.422303 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.422280 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:05.423704 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.423683 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-xks-ca" pod "de1de9e2-97ac-4a1c-9690-cfc619c9eca2" (UID: "de1de9e2-97ac-4a1c-9690-cfc619c9eca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:58:05.423792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.423723 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-kube-api-access-64jkr" pod "de1de9e2-97ac-4a1c-9690-cfc619c9eca2" (UID: "de1de9e2-97ac-4a1c-9690-cfc619c9eca2"). InnerVolumeSpecName "kube-api-access-64jkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:05.523977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.523946 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:05.523977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.523973 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jkr\" (UniqueName: \"kubernetes.io/projected/de1de9e2-97ac-4a1c-9690-cfc619c9eca2-kube-api-access-64jkr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:05.575754 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:05.575724 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:06.083713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:06.083681 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" Nov 01 22:58:06.083897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:06.083674 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt" event={"ID":"de1de9e2-97ac-4a1c-9690-cfc619c9eca2","Type":"ContainerDied","Data":"fb57aef44eed654753e4133e283dd7bb6f01bf56956cd3b2c8d2864513a594db"} Nov 01 22:58:06.083897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:06.083780 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb57aef44eed654753e4133e283dd7bb6f01bf56956cd3b2c8d2864513a594db" Nov 01 22:58:06.266096 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:06.266061 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:06.954586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:06.954543 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:07.554149 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:07.554110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:08.270326 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:08.270287 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:08.979120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:08.979085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:09.656923 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:09.656882 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:10.469971 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:10.469940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:11.492821 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:11.492791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:11.991263 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:11.991224 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f"] Nov 01 22:58:12.016888 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:12.016853 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb"] Nov 01 22:58:12.043409 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:12.043379 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 22:58:12.099766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:12.099731 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:12.227256 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:12.227216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:12.996200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:12.996164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:14.056467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:14.056427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:14.527772 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:58:14.527736 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:58:14.667108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:14.667078 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:14.970178 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:14.970144 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:15.908859 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:15.908827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:16.675116 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:16.675074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:16.992313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:16.992224 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" containerID="cri-o://85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7" gracePeriod=25 Nov 01 22:58:17.018313 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:17.018257 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" containerID="cri-o://0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564" gracePeriod=25 Nov 01 22:58:17.114030 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:17.113987 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:58:17.115161 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:17.115135 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:58:17.402032 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:17.401946 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:18.270530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:18.270497 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:19.134077 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:19.134043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:19.865364 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:19.865319 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:20.516670 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.516638 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw"] Nov 01 22:58:20.517907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.517881 4981 cpu_manager.go:414] "containerMap: removing container" podUID="de1de9e2-97ac-4a1c-9690-cfc619c9eca2" containerName="revoke-keys" Nov 01 22:58:20.517907 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.517905 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de1de9e2-97ac-4a1c-9690-cfc619c9eca2" containerName="revoke-keys" Nov 01 22:58:20.518168 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.518154 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="de1de9e2-97ac-4a1c-9690-cfc619c9eca2" containerName="revoke-keys" Nov 01 22:58:20.522732 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.522710 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.537324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.537301 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d73b25-serving-cert\"" Nov 01 22:58:20.543601 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.543555 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw"] Nov 01 22:58:20.570324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.570282 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:20.667651 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.667615 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba856d5-aa40-41f9-bdbd-ab43935f23fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.667825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.667758 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.667936 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.667921 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.668035 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.668021 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkhb\" (UniqueName: \"kubernetes.io/projected/fba856d5-aa40-41f9-bdbd-ab43935f23fa-kube-api-access-nwkhb\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.670218 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.670196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:20.771108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.770947 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.774347 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.774317 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkhb\" (UniqueName: \"kubernetes.io/projected/fba856d5-aa40-41f9-bdbd-ab43935f23fa-kube-api-access-nwkhb\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.775191 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.775164 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba856d5-aa40-41f9-bdbd-ab43935f23fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.775864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.775640 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.784818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.776762 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba856d5-aa40-41f9-bdbd-ab43935f23fa-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.784818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.778084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:20.784818 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.783114 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.790214 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.790190 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.791283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.791262 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkhb\" (UniqueName: \"kubernetes.io/projected/fba856d5-aa40-41f9-bdbd-ab43935f23fa-kube-api-access-nwkhb\") pod \"maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:20.842699 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:20.842659 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:21.274597 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:21.273760 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw"] Nov 01 22:58:21.342928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:21.342889 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" event={"ID":"fba856d5-aa40-41f9-bdbd-ab43935f23fa","Type":"ContainerStarted","Data":"659107c846aecbd2f55ebea629aaebfa376a23858cd8e32bb9fb72d69d22c494"} Nov 01 22:58:21.465249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:21.465216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:22.241308 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:22.241275 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:22.368319 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:22.368277 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" event={"ID":"fba856d5-aa40-41f9-bdbd-ab43935f23fa","Type":"ContainerStarted","Data":"e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430"} Nov 01 22:58:22.368797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:22.368365 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:22.393429 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:22.393380 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" podStartSLOduration=1.840653382 podStartE2EDuration="2.3933667s" totalImagesPullingTime="552.713318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:22.388338727 +0000 UTC m=+2067.816216658" watchObservedRunningTime="2026-11-01 22:58:22.3933667 +0000 UTC m=+2067.821244630" Nov 01 22:58:23.051114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:23.051079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:23.656811 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:23.656766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:24.226469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:24.226436 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw"] Nov 01 22:58:24.226801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:24.226762 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" containerID="cri-o://8ec0664be02366b1269f59dc80dcf0d8d1461bb5fe0e8246585a5e1e6c575f04" gracePeriod=30 Nov 01 22:58:24.233163 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:24.233135 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7"] Nov 01 22:58:24.233394 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:24.233369 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" containerID="cri-o://157002ec4862ce6d58b450b996ff8e9a4eafe80f72408c52f3a6f2871e390e0b" gracePeriod=30 Nov 01 22:58:24.239518 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:24.239496 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7"] Nov 01 22:58:24.239750 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:24.239726 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" containerID="cri-o://3c526cefccca3201376b767ead359b6ae9d729563cf4e3c0ae20772c8e45668f" gracePeriod=30 Nov 01 22:58:24.322363 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:24.322323 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:25.012121 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:25.012086 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:25.021285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:25.021256 4981 patch_prober.go:29] interesting pod/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:58:25.021434 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:25.021310 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:58:25.659441 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:25.659405 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:26.372734 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:26.372703 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:27.114148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:27.114108 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:58:27.115230 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:27.115194 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 22:58:27.255209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:27.255177 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:28.098606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:28.081305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:28.383467 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:28.383378 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 22:58:28.672662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:28.672632 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:28.856227 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:28.853863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:29.185522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.185484 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:29.296287 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.296108 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt"] Nov 01 22:58:29.319800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.319765 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b78bd-rnwtt"] Nov 01 22:58:29.494797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.494708 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:29.508802 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.508766 4981 generic.go:356] "Generic (PLEG): container finished" podID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerID="157002ec4862ce6d58b450b996ff8e9a4eafe80f72408c52f3a6f2871e390e0b" exitCode=0 Nov 01 22:58:29.509037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.509013 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" event={"ID":"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40","Type":"ContainerDied","Data":"157002ec4862ce6d58b450b996ff8e9a4eafe80f72408c52f3a6f2871e390e0b"} Nov 01 22:58:29.512136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.512112 4981 generic.go:356] "Generic (PLEG): container finished" podID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerID="8ec0664be02366b1269f59dc80dcf0d8d1461bb5fe0e8246585a5e1e6c575f04" exitCode=0 Nov 01 22:58:29.512268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.512186 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" event={"ID":"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61","Type":"ContainerDied","Data":"8ec0664be02366b1269f59dc80dcf0d8d1461bb5fe0e8246585a5e1e6c575f04"} Nov 01 22:58:29.512268 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.512248 4981 scope.go:122] "RemoveContainer" containerID="cb9dccbfbac2618181f24c84af91df0b6beecd726fe630392cf5847807901a4d" Nov 01 22:58:29.515069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.515049 4981 generic.go:356] "Generic (PLEG): container finished" podID="995ecca3-6286-489f-ba37-a6170687b0c3" containerID="3c526cefccca3201376b767ead359b6ae9d729563cf4e3c0ae20772c8e45668f" exitCode=0 Nov 01 22:58:29.515176 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.515077 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" event={"ID":"995ecca3-6286-489f-ba37-a6170687b0c3","Type":"ContainerDied","Data":"3c526cefccca3201376b767ead359b6ae9d729563cf4e3c0ae20772c8e45668f"} Nov 01 22:58:29.619535 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.619059 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:58:29.646723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.645878 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-socket\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.646723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.645954 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995ecca3-6286-489f-ba37-a6170687b0c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995ecca3-6286-489f-ba37-a6170687b0c3-istiod-ca-cert\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.646723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.646199 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-socket" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.646723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.646373 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995ecca3-6286-489f-ba37-a6170687b0c3-istiod-ca-cert" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:29.646723 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.646462 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-istio-token\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-istio-token\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.648743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.648717 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/995ecca3-6286-489f-ba37-a6170687b0c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995ecca3-6286-489f-ba37-a6170687b0c3-istio-podinfo\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.648868 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.648820 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-kube-api-access-q6n6z\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-kube-api-access-q6n6z\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.648954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.648929 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-data\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.649051 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.649036 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-credential-socket\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.649208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.649189 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-certs\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.649278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.649242 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-envoy\") pod \"995ecca3-6286-489f-ba37-a6170687b0c3\" (UID: \"995ecca3-6286-489f-ba37-a6170687b0c3\") " Nov 01 22:58:29.657208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.651991 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-certs" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.657208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.652392 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.657208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.652411 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.657208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.652422 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/995ecca3-6286-489f-ba37-a6170687b0c3-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.657208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.652730 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-credential-socket" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.657208 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.653287 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-data" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.658304 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.658273 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-istio-token" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:29.658729 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.658702 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-kube-api-access-q6n6z" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "kube-api-access-q6n6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:29.661114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.661091 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-envoy" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.661647 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.661621 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/995ecca3-6286-489f-ba37-a6170687b0c3-istio-podinfo" pod "995ecca3-6286-489f-ba37-a6170687b0c3" (UID: "995ecca3-6286-489f-ba37-a6170687b0c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:29.673772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.673751 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:58:29.688490 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.688461 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:58:29.704820 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.704792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:29.753715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753639 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-credential-socket\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.753715 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753682 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-kube-api-access-xpwj4\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-kube-api-access-xpwj4\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.753965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753729 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-certs\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.753965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753761 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istiod-ca-cert\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.753965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753815 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-podinfo\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.753965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753880 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-podinfo\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.753965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753910 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-data\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.753965 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.753942 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-token\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-token\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.754307 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.754030 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-token\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-token\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.754603 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.754555 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-certs" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.754720 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.754695 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-data" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.754787 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.754752 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-credential-socket\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.754844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.754808 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-socket\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.754928 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.754905 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-data\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.754996 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.754974 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istiod-ca-cert\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.755055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755003 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-certs\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.755055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755027 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-credential-socket" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.755055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755043 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istiod-ca-cert" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:29.755229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755034 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-socket\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.755229 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755221 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-socket" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.755344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755224 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-data" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.755344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755242 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-envoy\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.755477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755453 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-kube-api-access-9lfzr\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-kube-api-access-9lfzr\") pod \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\" (UID: \"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61\") " Nov 01 22:58:29.755543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755523 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istiod-ca-cert" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:29.755677 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755655 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-envoy\") pod \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\" (UID: \"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40\") " Nov 01 22:58:29.755749 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755692 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-credential-socket" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.755895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755866 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-certs" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.755895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.755889 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-socket" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756529 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756556 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756585 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756601 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756617 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/995ecca3-6286-489f-ba37-a6170687b0c3-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756630 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756645 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6n6z\" (UniqueName: \"kubernetes.io/projected/995ecca3-6286-489f-ba37-a6170687b0c3-kube-api-access-q6n6z\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756659 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756673 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756688 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756702 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756715 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756728 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756742 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756755 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.756877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.756771 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/995ecca3-6286-489f-ba37-a6170687b0c3-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.758269 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.758245 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-podinfo" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:29.758388 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.758361 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-envoy" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.758569 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.758545 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-kube-api-access-xpwj4" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "kube-api-access-xpwj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:29.760037 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.760009 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-token" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:29.760226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.760204 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-token" pod "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" (UID: "0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:29.760348 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.760330 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-podinfo" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:29.760695 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.760668 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-kube-api-access-9lfzr" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "kube-api-access-9lfzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:29.762789 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.762768 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-envoy" pod "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" (UID: "b000cee6-3d32-42de-ad5d-fa3d0e4fbb61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:29.858274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858241 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.858274 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858277 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lfzr\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-kube-api-access-9lfzr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.858477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858295 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.858477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858311 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwj4\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-kube-api-access-xpwj4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.858477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858325 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.858477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858340 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.858477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858353 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:29.858477 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:29.858368 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:30.447135 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.447105 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:30.540114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.540014 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" Nov 01 22:58:30.540114 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.540041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7" event={"ID":"995ecca3-6286-489f-ba37-a6170687b0c3","Type":"ContainerDied","Data":"23c2263577960156eff1a13d4b080ed24e1b9289c58ec3b37fe7c4b98084ebe5"} Nov 01 22:58:30.540391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.540119 4981 scope.go:122] "RemoveContainer" containerID="3c526cefccca3201376b767ead359b6ae9d729563cf4e3c0ae20772c8e45668f" Nov 01 22:58:30.548856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.548834 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" Nov 01 22:58:30.549003 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.548868 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7" event={"ID":"0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40","Type":"ContainerDied","Data":"1613394276c2f8e2043207a69fa1c6818a98163bd9b2b650c1c40cd338b09925"} Nov 01 22:58:30.551285 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.551260 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" event={"ID":"b000cee6-3d32-42de-ad5d-fa3d0e4fbb61","Type":"ContainerDied","Data":"dc6f47dcdc81ddbeed3dcc452be36501e0262e69561fc3eb53f3bfba809a2bd1"} Nov 01 22:58:30.551386 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.551315 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw" Nov 01 22:58:30.555954 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.555934 4981 scope.go:122] "RemoveContainer" containerID="157002ec4862ce6d58b450b996ff8e9a4eafe80f72408c52f3a6f2871e390e0b" Nov 01 22:58:30.571118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.571097 4981 scope.go:122] "RemoveContainer" containerID="8ec0664be02366b1269f59dc80dcf0d8d1461bb5fe0e8246585a5e1e6c575f04" Nov 01 22:58:30.574232 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.574211 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7"] Nov 01 22:58:30.579556 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.579531 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-7rdj7"] Nov 01 22:58:30.590684 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.590657 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw"] Nov 01 22:58:30.595883 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.595856 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-n7dbw"] Nov 01 22:58:30.605660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.605631 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7"] Nov 01 22:58:30.612421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.612395 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-jxmh7"] Nov 01 22:58:30.737652 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.737622 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" path="/var/lib/kubelet/pods/0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40/volumes" Nov 01 22:58:30.738180 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.738165 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" path="/var/lib/kubelet/pods/995ecca3-6286-489f-ba37-a6170687b0c3/volumes" Nov 01 22:58:30.738658 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.738639 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" path="/var/lib/kubelet/pods/b000cee6-3d32-42de-ad5d-fa3d0e4fbb61/volumes" Nov 01 22:58:30.739414 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:30.739399 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de1de9e2-97ac-4a1c-9690-cfc619c9eca2" path="/var/lib/kubelet/pods/de1de9e2-97ac-4a1c-9690-cfc619c9eca2/volumes" Nov 01 22:58:31.244815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:31.244775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:32.086174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:32.086136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:32.700880 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:32.700844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:33.343929 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:33.343894 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:34.030254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:34.030224 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:34.782344 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:34.767081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:35.389334 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:35.389306 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:35.829262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:35.829222 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:35.932204 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:35.932171 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:36.134662 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.134558 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99"] Nov 01 22:58:36.134903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.134873 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" containerID="cri-o://68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141" gracePeriod=30 Nov 01 22:58:36.161417 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.161391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:36.326112 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.326068 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:36.759892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.759865 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx"] Nov 01 22:58:36.760726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760710 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" Nov 01 22:58:36.760785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760726 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" Nov 01 22:58:36.760785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760749 4981 cpu_manager.go:414] "containerMap: removing container" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" Nov 01 22:58:36.760785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760754 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" Nov 01 22:58:36.760908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760786 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" Nov 01 22:58:36.760908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760792 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" Nov 01 22:58:36.760908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760800 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" Nov 01 22:58:36.760908 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760805 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" Nov 01 22:58:36.761039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760941 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" Nov 01 22:58:36.761039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760952 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="995ecca3-6286-489f-ba37-a6170687b0c3" containerName="istio-proxy" Nov 01 22:58:36.761039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760967 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b000cee6-3d32-42de-ad5d-fa3d0e4fbb61" containerName="istio-proxy" Nov 01 22:58:36.761039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.760973 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0e41e5-83ae-42ed-8e29-fb1a3f1c1a40" containerName="istio-proxy" Nov 01 22:58:36.767370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.767351 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:36.774870 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.774851 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:58:36.776293 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.775822 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:58:36.783713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.783691 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx"] Nov 01 22:58:36.905312 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.905278 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:36.905726 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.905546 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:36.905785 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:36.905770 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcvxx\" (UniqueName: \"kubernetes.io/projected/852babd9-bf68-4fd6-9dab-a6251fdcc30f-kube-api-access-xcvxx\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.007903 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.007870 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.008120 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.008101 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcvxx\" (UniqueName: \"kubernetes.io/projected/852babd9-bf68-4fd6-9dab-a6251fdcc30f-kube-api-access-xcvxx\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.008192 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.008176 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.008609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.008591 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.010660 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.010613 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.018169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.018144 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcvxx\" (UniqueName: \"kubernetes.io/projected/852babd9-bf68-4fd6-9dab-a6251fdcc30f-kube-api-access-xcvxx\") pod \"maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.086056 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.086027 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:37.113297 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:58:37.113275 4981 logging.go:55] [core] [Channel #8743 SubChannel #8744]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.211:50052", ServerName: "10.134.0.211:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.211:50052: connect: connection refused" Nov 01 22:58:37.114732 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:58:37.114716 4981 logging.go:55] [core] [Channel #8745 SubChannel #8746]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.210:50052", ServerName: "10.134.0.210:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.210:50052: connect: connection refused" Nov 01 22:58:37.380512 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:58:37.380480 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852babd9_bf68_4fd6_9dab_a6251fdcc30f.slice/crio-c965e18ba205f1a344ee4d1921fdf50c2d58f628b51e78675a56e3988b2a429e WatchSource:0}: Error finding container c965e18ba205f1a344ee4d1921fdf50c2d58f628b51e78675a56e3988b2a429e: Status 404 returned error can't find the container with id c965e18ba205f1a344ee4d1921fdf50c2d58f628b51e78675a56e3988b2a429e Nov 01 22:58:37.382119 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.382094 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx"] Nov 01 22:58:37.394948 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.394925 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:58:37.397559 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.397541 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:58:37.519324 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.519291 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f207d6-ed80-4b74-a330-57b836b5fba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f207d6-ed80-4b74-a330-57b836b5fba5-plugins-config-volume\") pod \"15f207d6-ed80-4b74-a330-57b836b5fba5\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " Nov 01 22:58:37.519495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.519449 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f207d6-ed80-4b74-a330-57b836b5fba5-kube-api-access-l5h95\" (UniqueName: \"kubernetes.io/projected/15f207d6-ed80-4b74-a330-57b836b5fba5-kube-api-access-l5h95\") pod \"15f207d6-ed80-4b74-a330-57b836b5fba5\" (UID: \"15f207d6-ed80-4b74-a330-57b836b5fba5\") " Nov 01 22:58:37.519495 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.519474 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb05797-6bbf-45c1-a341-088aeadb04e0-kube-api-access-dz62b\" (UniqueName: \"kubernetes.io/projected/2eb05797-6bbf-45c1-a341-088aeadb04e0-kube-api-access-dz62b\") pod \"2eb05797-6bbf-45c1-a341-088aeadb04e0\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " Nov 01 22:58:37.519609 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.519563 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eb05797-6bbf-45c1-a341-088aeadb04e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb05797-6bbf-45c1-a341-088aeadb04e0-plugins-config-volume\") pod \"2eb05797-6bbf-45c1-a341-088aeadb04e0\" (UID: \"2eb05797-6bbf-45c1-a341-088aeadb04e0\") " Nov 01 22:58:37.519753 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.519726 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f207d6-ed80-4b74-a330-57b836b5fba5-plugins-config-volume" pod "15f207d6-ed80-4b74-a330-57b836b5fba5" (UID: "15f207d6-ed80-4b74-a330-57b836b5fba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:37.520044 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.520019 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb05797-6bbf-45c1-a341-088aeadb04e0-plugins-config-volume" pod "2eb05797-6bbf-45c1-a341-088aeadb04e0" (UID: "2eb05797-6bbf-45c1-a341-088aeadb04e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:37.520502 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.520483 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f207d6-ed80-4b74-a330-57b836b5fba5-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:37.520644 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.520504 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb05797-6bbf-45c1-a341-088aeadb04e0-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:37.521758 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.521737 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb05797-6bbf-45c1-a341-088aeadb04e0-kube-api-access-dz62b" pod "2eb05797-6bbf-45c1-a341-088aeadb04e0" (UID: "2eb05797-6bbf-45c1-a341-088aeadb04e0"). InnerVolumeSpecName "kube-api-access-dz62b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:37.521853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.521766 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f207d6-ed80-4b74-a330-57b836b5fba5-kube-api-access-l5h95" pod "15f207d6-ed80-4b74-a330-57b836b5fba5" (UID: "15f207d6-ed80-4b74-a330-57b836b5fba5"). InnerVolumeSpecName "kube-api-access-l5h95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:37.545792 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.545769 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:37.626207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.626155 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5h95\" (UniqueName: \"kubernetes.io/projected/15f207d6-ed80-4b74-a330-57b836b5fba5-kube-api-access-l5h95\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:37.626207 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.626201 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz62b\" (UniqueName: \"kubernetes.io/projected/2eb05797-6bbf-45c1-a341-088aeadb04e0-kube-api-access-dz62b\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:37.654219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.654185 4981 generic.go:356] "Generic (PLEG): container finished" podID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerID="85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7" exitCode=0 Nov 01 22:58:37.654377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.654232 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" Nov 01 22:58:37.654377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.654260 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" event={"ID":"15f207d6-ed80-4b74-a330-57b836b5fba5","Type":"ContainerDied","Data":"85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7"} Nov 01 22:58:37.654377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.654314 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" event={"ID":"15f207d6-ed80-4b74-a330-57b836b5fba5","Type":"ContainerDied","Data":"b7ca5dcc747c8c5f82bf3069a335de39703ed5c3424e4e8b997558218d5e208b"} Nov 01 22:58:37.654377 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.654342 4981 scope.go:122] "RemoveContainer" containerID="85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7" Nov 01 22:58:37.655925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.655901 4981 generic.go:356] "Generic (PLEG): container finished" podID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerID="0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564" exitCode=0 Nov 01 22:58:37.656027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.655953 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" Nov 01 22:58:37.656027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.655986 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" event={"ID":"2eb05797-6bbf-45c1-a341-088aeadb04e0","Type":"ContainerDied","Data":"0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564"} Nov 01 22:58:37.656143 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.656030 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" event={"ID":"2eb05797-6bbf-45c1-a341-088aeadb04e0","Type":"ContainerDied","Data":"1380d829cdda838ba3ac1a34583b28e2e14d6d75d97813f9713539c7d622654e"} Nov 01 22:58:37.665240 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.665215 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" event={"ID":"852babd9-bf68-4fd6-9dab-a6251fdcc30f","Type":"ContainerStarted","Data":"c965e18ba205f1a344ee4d1921fdf50c2d58f628b51e78675a56e3988b2a429e"} Nov 01 22:58:37.670989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.670490 4981 scope.go:122] "RemoveContainer" containerID="85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7" Nov 01 22:58:37.670989 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:58:37.670875 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7\": container with ID starting with 85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7 not found: ID does not exist" containerID="85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7" Nov 01 22:58:37.670989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.670909 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7"} err="failed to get container status \"85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7\": rpc error: code = NotFound desc = could not find container \"85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7\": container with ID starting with 85574b1189ca9eea002c51bab83349a73778b24df9b68200269f23978f25ecb7 not found: ID does not exist" Nov 01 22:58:37.670989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.670930 4981 scope.go:122] "RemoveContainer" containerID="0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564" Nov 01 22:58:37.685047 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.685028 4981 scope.go:122] "RemoveContainer" containerID="0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564" Nov 01 22:58:37.685326 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:58:37.685292 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564\": container with ID starting with 0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564 not found: ID does not exist" containerID="0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564" Nov 01 22:58:37.685432 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.685327 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564"} err="failed to get container status \"0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564\": rpc error: code = NotFound desc = could not find container \"0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564\": container with ID starting with 0aaedbf8fa5973b8a51a195f316a8f4720a290f10bee70f48349d8ed492c1564 not found: ID does not exist" Nov 01 22:58:37.686914 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.686898 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f"] Nov 01 22:58:37.691709 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.691690 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f"] Nov 01 22:58:37.697937 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.697918 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb"] Nov 01 22:58:37.704351 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:37.704325 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb"] Nov 01 22:58:38.113752 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.113716 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6b78bd-7cb8c4d4cf-xtg2f" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.211:50052\" within 1s: context deadline exceeded" Nov 01 22:58:38.114828 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.114803 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6b78bd-66687fb55f-6rphb" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.210:50052\" within 1s: context deadline exceeded" Nov 01 22:58:38.257838 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.257803 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:38.677138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.677104 4981 generic.go:356] "Generic (PLEG): container finished" podID="852babd9-bf68-4fd6-9dab-a6251fdcc30f" containerID="ce3d3ffcb13ed5807ded4661ca543863ae443a764f57d51fb768e6456519e119" exitCode=0 Nov 01 22:58:38.677318 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.677195 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" event={"ID":"852babd9-bf68-4fd6-9dab-a6251fdcc30f","Type":"ContainerDied","Data":"ce3d3ffcb13ed5807ded4661ca543863ae443a764f57d51fb768e6456519e119"} Nov 01 22:58:38.739517 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.739488 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" path="/var/lib/kubelet/pods/15f207d6-ed80-4b74-a330-57b836b5fba5/volumes" Nov 01 22:58:38.739950 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.739934 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" path="/var/lib/kubelet/pods/2eb05797-6bbf-45c1-a341-088aeadb04e0/volumes" Nov 01 22:58:38.850844 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:38.850815 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:39.487512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.487469 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:39.882522 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.882497 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:39.968081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.968049 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852babd9-bf68-4fd6-9dab-a6251fdcc30f-kube-api-access-xcvxx\" (UniqueName: \"kubernetes.io/projected/852babd9-bf68-4fd6-9dab-a6251fdcc30f-kube-api-access-xcvxx\") pod \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " Nov 01 22:58:39.968253 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.968177 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-openshift-ca\") pod \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " Nov 01 22:58:39.968352 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.968336 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-xks-ca\") pod \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\" (UID: \"852babd9-bf68-4fd6-9dab-a6251fdcc30f\") " Nov 01 22:58:39.968566 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.968542 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-openshift-ca" pod "852babd9-bf68-4fd6-9dab-a6251fdcc30f" (UID: "852babd9-bf68-4fd6-9dab-a6251fdcc30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:39.968953 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.968930 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:39.970201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.970176 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852babd9-bf68-4fd6-9dab-a6251fdcc30f-kube-api-access-xcvxx" pod "852babd9-bf68-4fd6-9dab-a6251fdcc30f" (UID: "852babd9-bf68-4fd6-9dab-a6251fdcc30f"). InnerVolumeSpecName "kube-api-access-xcvxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:39.970346 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:39.970330 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-xks-ca" pod "852babd9-bf68-4fd6-9dab-a6251fdcc30f" (UID: "852babd9-bf68-4fd6-9dab-a6251fdcc30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:58:40.070530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:40.070446 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852babd9-bf68-4fd6-9dab-a6251fdcc30f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:40.070530 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:40.070476 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcvxx\" (UniqueName: \"kubernetes.io/projected/852babd9-bf68-4fd6-9dab-a6251fdcc30f-kube-api-access-xcvxx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:40.208124 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:40.208093 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:40.695605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:40.695562 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" Nov 01 22:58:40.695605 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:40.695563 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx" event={"ID":"852babd9-bf68-4fd6-9dab-a6251fdcc30f","Type":"ContainerDied","Data":"c965e18ba205f1a344ee4d1921fdf50c2d58f628b51e78675a56e3988b2a429e"} Nov 01 22:58:40.696027 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:40.695660 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c965e18ba205f1a344ee4d1921fdf50c2d58f628b51e78675a56e3988b2a429e" Nov 01 22:58:40.842223 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:40.842166 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:41.429621 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.429591 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:58:41.577082 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.577050 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595367 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-socket\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595608 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-data\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595731 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-istio-token\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-istio-token\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595793 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-credential-socket\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595820 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63056725-77e3-4c7b-8132-1ec6325a1081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63056725-77e3-4c7b-8132-1ec6325a1081-istio-podinfo\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595858 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-kube-api-access-ll754\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-kube-api-access-ll754\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595919 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63056725-77e3-4c7b-8132-1ec6325a1081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63056725-77e3-4c7b-8132-1ec6325a1081-istiod-ca-cert\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.595949 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-envoy\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.596120 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-certs\") pod \"63056725-77e3-4c7b-8132-1ec6325a1081\" (UID: \"63056725-77e3-4c7b-8132-1ec6325a1081\") " Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.597153 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-certs" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.597384 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-socket" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.597539 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-data" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.598886 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63056725-77e3-4c7b-8132-1ec6325a1081-istiod-ca-cert" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:58:41.601610 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.600323 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-credential-socket" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.673946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.652466 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-istio-token" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.673946 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.657369 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-kube-api-access-ll754" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "kube-api-access-ll754". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:58:41.685034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.684064 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63056725-77e3-4c7b-8132-1ec6325a1081-istio-podinfo" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:58:41.685034 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.684166 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-envoy" pod "63056725-77e3-4c7b-8132-1ec6325a1081" (UID: "63056725-77e3-4c7b-8132-1ec6325a1081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:58:41.698993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.698958 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.698993 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.698994 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.699617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.699009 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.699617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.699024 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.699617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.699039 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.699617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.699052 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63056725-77e3-4c7b-8132-1ec6325a1081-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.699617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.699066 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll754\" (UniqueName: \"kubernetes.io/projected/63056725-77e3-4c7b-8132-1ec6325a1081-kube-api-access-ll754\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.699617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.699080 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63056725-77e3-4c7b-8132-1ec6325a1081-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.699617 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.699093 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63056725-77e3-4c7b-8132-1ec6325a1081-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:58:41.713336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.713308 4981 generic.go:356] "Generic (PLEG): container finished" podID="63056725-77e3-4c7b-8132-1ec6325a1081" containerID="68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141" exitCode=0 Nov 01 22:58:41.713521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.713360 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" Nov 01 22:58:41.713521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.713349 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" event={"ID":"63056725-77e3-4c7b-8132-1ec6325a1081","Type":"ContainerDied","Data":"68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141"} Nov 01 22:58:41.713521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.713490 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99" event={"ID":"63056725-77e3-4c7b-8132-1ec6325a1081","Type":"ContainerDied","Data":"50111f0636e874ac64367afbb998b5b80d9365dd54c131167755bfdc593e9692"} Nov 01 22:58:41.713521 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.713515 4981 scope.go:122] "RemoveContainer" containerID="68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141" Nov 01 22:58:41.741118 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.741087 4981 scope.go:122] "RemoveContainer" containerID="68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141" Nov 01 22:58:41.741803 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:58:41.741532 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141\": container with ID starting with 68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141 not found: ID does not exist" containerID="68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141" Nov 01 22:58:41.741803 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.741595 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141"} err="failed to get container status \"68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141\": rpc error: code = NotFound desc = could not find container \"68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141\": container with ID starting with 68cdaf7919d6cbd21775cea1cc427d05d42deac0021735fffdb052035810f141 not found: ID does not exist" Nov 01 22:58:41.747682 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.747612 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99"] Nov 01 22:58:41.755639 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:41.755618 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b78bd-openshift-default-8795868d5-w6v99"] Nov 01 22:58:42.464886 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:42.464809 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:42.737142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:42.737056 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" path="/var/lib/kubelet/pods/63056725-77e3-4c7b-8132-1ec6325a1081/volumes" Nov 01 22:58:43.239087 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:43.239051 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:43.843203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:43.843173 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:43.947298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:43.947266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:44.527267 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:58:44.527233 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:58:44.656476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:44.656449 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:45.539136 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:45.539095 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:46.152165 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:46.152134 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:46.775078 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:46.775050 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:46.977952 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:46.977919 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:47.080822 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:47.080740 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:47.522607 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:47.521551 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:47.749458 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:47.749427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:48.421884 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:48.421850 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:49.365570 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:49.365534 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:50.061225 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:50.061194 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:50.759329 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:50.759302 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:51.486190 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:51.486158 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:52.242867 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:52.242831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:52.963567 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:52.963537 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:53.744895 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:53.744862 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:54.445130 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:54.445102 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:55.139558 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:55.139527 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:55.867661 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:55.867626 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:56.651156 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:56.651126 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:57.352632 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:57.352596 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:58.106464 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:58.106433 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:58.962636 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:58.962594 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:58:59.710241 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:58:59.710207 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:00.373069 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:00.373042 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:00.928623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:00.928590 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:01.132453 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:01.132415 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:01.226673 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:01.221970 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:01.321413 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:01.321341 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx"] Nov 01 22:59:01.350200 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:01.350126 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0e685d-g8ljx"] Nov 01 22:59:01.443672 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:01.443638 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:01.710382 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:01.710344 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:02.268476 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:02.268442 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:02.748107 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:02.748064 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852babd9-bf68-4fd6-9dab-a6251fdcc30f" path="/var/lib/kubelet/pods/852babd9-bf68-4fd6-9dab-a6251fdcc30f/volumes" Nov 01 22:59:03.026616 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:03.026533 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:03.765797 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:03.765763 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:04.382534 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:04.382468 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:05.086698 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:05.086667 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:05.759278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:05.759246 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:06.558113 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:06.558078 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:07.247707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:07.247671 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:07.734986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:07.734945 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:07.821390 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:07.821355 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:08.046158 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:08.046071 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr"] Nov 01 22:59:08.046772 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:08.046719 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" containerID="cri-o://1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680" gracePeriod=30 Nov 01 22:59:08.226335 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:08.226299 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:08.365892 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:08.365730 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:08.454925 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:08.454895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:08.855081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:08.855052 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:09.893049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:09.893018 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:10.789969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:10.789939 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:10.881153 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:10.881122 4981 patch_prober.go:29] interesting pod/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 22:59:10.881337 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:10.881195 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 22:59:11.573654 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:11.573619 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:12.660497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:12.660463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:13.352779 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.352753 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:59:13.467339 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467307 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-data\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.467623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467482 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08a8bacc-1268-463d-b491-b9a705807834-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a8bacc-1268-463d-b491-b9a705807834-istio-podinfo\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.467623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467527 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-socket\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.467623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467567 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-certs\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.467623 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467604 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-data" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.467801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467654 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-istio-token\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-istio-token\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.467801 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467697 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-envoy\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.467917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467848 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-socket" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.467917 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467904 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a8bacc-1268-463d-b491-b9a705807834-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a8bacc-1268-463d-b491-b9a705807834-istiod-ca-cert\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.468036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467940 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-certs" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.468036 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.467949 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-kube-api-access-c9c89\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-kube-api-access-c9c89\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.468141 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468045 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-credential-socket\") pod \"08a8bacc-1268-463d-b491-b9a705807834\" (UID: \"08a8bacc-1268-463d-b491-b9a705807834\") " Nov 01 22:59:13.468336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468310 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a8bacc-1268-463d-b491-b9a705807834-istiod-ca-cert" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:13.468336 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468314 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-credential-socket" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.468763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468745 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.468763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468763 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.468913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468772 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08a8bacc-1268-463d-b491-b9a705807834-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.468913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468781 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.468913 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.468790 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.470002 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.469977 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-istio-token" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:13.470111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.470043 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-kube-api-access-c9c89" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "kube-api-access-c9c89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:13.470174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.470146 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08a8bacc-1268-463d-b491-b9a705807834-istio-podinfo" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 22:59:13.470322 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.470305 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-envoy" pod "08a8bacc-1268-463d-b491-b9a705807834" (UID: "08a8bacc-1268-463d-b491-b9a705807834"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 22:59:13.479986 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.479961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:13.570735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.570656 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08a8bacc-1268-463d-b491-b9a705807834-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.570735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.570684 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.570735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.570694 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08a8bacc-1268-463d-b491-b9a705807834-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:13.570735 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:13.570705 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9c89\" (UniqueName: \"kubernetes.io/projected/08a8bacc-1268-463d-b491-b9a705807834-kube-api-access-c9c89\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:14.103690 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.103635 4981 generic.go:356] "Generic (PLEG): container finished" podID="08a8bacc-1268-463d-b491-b9a705807834" containerID="1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680" exitCode=0 Nov 01 22:59:14.104222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.103714 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" Nov 01 22:59:14.104222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.103723 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" event={"ID":"08a8bacc-1268-463d-b491-b9a705807834","Type":"ContainerDied","Data":"1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680"} Nov 01 22:59:14.104222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.103797 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr" event={"ID":"08a8bacc-1268-463d-b491-b9a705807834","Type":"ContainerDied","Data":"6dcac34cf36629d8bdec6a47a0efa49e08a222aaf03bcbb4921be1044d171fa7"} Nov 01 22:59:14.104222 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.103833 4981 scope.go:122] "RemoveContainer" containerID="1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680" Nov 01 22:59:14.117244 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.117227 4981 scope.go:122] "RemoveContainer" containerID="7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e" Nov 01 22:59:14.131137 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.131120 4981 scope.go:122] "RemoveContainer" containerID="1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680" Nov 01 22:59:14.131395 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:59:14.131375 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680\": container with ID starting with 1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680 not found: ID does not exist" containerID="1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680" Nov 01 22:59:14.131455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.131403 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680"} err="failed to get container status \"1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680\": rpc error: code = NotFound desc = could not find container \"1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680\": container with ID starting with 1b5b6972db57acfdacbd7a52cd13c23e493d214a206ccffc982cba4ba8f8b680 not found: ID does not exist" Nov 01 22:59:14.131455 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.131419 4981 scope.go:122] "RemoveContainer" containerID="7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e" Nov 01 22:59:14.131753 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:59:14.131733 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e\": container with ID starting with 7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e not found: ID does not exist" containerID="7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e" Nov 01 22:59:14.131819 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.131758 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e"} err="failed to get container status \"7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e\": rpc error: code = NotFound desc = could not find container \"7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e\": container with ID starting with 7474ace97ca9d7a8685902979523798cf33a80b23439ec6de0281ccef38c764e not found: ID does not exist" Nov 01 22:59:14.134742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.134720 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr"] Nov 01 22:59:14.139138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.139118 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0e685d-openshift-default-b9b7b778f-ch9qr"] Nov 01 22:59:14.260298 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.260267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:14.526430 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:59:14.526396 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:59:14.725826 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.725755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:14.738766 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:14.738741 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a8bacc-1268-463d-b491-b9a705807834" path="/var/lib/kubelet/pods/08a8bacc-1268-463d-b491-b9a705807834/volumes" Nov 01 22:59:15.162815 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:15.162739 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:15.343714 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:15.343677 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:16.129974 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:16.129945 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:17.299782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:17.299747 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:18.745457 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:18.745421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:19.620067 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:19.620033 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:20.588707 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:20.588676 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:21.580101 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:21.580072 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:22.522773 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:22.522745 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:23.277257 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:23.277212 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:23.581994 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:23.581894 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:24.243021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:24.242970 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:24.951620 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:24.951591 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:25.766775 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:25.766743 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:26.463494 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:26.463465 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:27.053142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:27.053111 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:28.078969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:28.078916 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:28.471264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:28.471228 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:29.024193 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:29.024159 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:29.659786 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:29.659755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:30.465155 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:30.465117 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:31.296465 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:31.296432 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:32.065283 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:32.065249 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:32.887235 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:32.887203 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:33.437203 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.437166 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp"] Nov 01 22:59:33.438142 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438119 4981 cpu_manager.go:414] "containerMap: removing container" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" Nov 01 22:59:33.438209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438145 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" Nov 01 22:59:33.438209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438174 4981 cpu_manager.go:414] "containerMap: removing container" podUID="852babd9-bf68-4fd6-9dab-a6251fdcc30f" containerName="revoke-keys" Nov 01 22:59:33.438209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438180 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852babd9-bf68-4fd6-9dab-a6251fdcc30f" containerName="revoke-keys" Nov 01 22:59:33.438209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438192 4981 cpu_manager.go:414] "containerMap: removing container" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" Nov 01 22:59:33.438209 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438197 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" Nov 01 22:59:33.438370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438213 4981 cpu_manager.go:414] "containerMap: removing container" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" Nov 01 22:59:33.438370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438218 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" Nov 01 22:59:33.438370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438238 4981 cpu_manager.go:414] "containerMap: removing container" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" Nov 01 22:59:33.438370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438243 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" Nov 01 22:59:33.438370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438252 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" Nov 01 22:59:33.438370 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438257 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" Nov 01 22:59:33.438560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438399 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f207d6-ed80-4b74-a330-57b836b5fba5" containerName="payload-processing" Nov 01 22:59:33.438560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438412 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="852babd9-bf68-4fd6-9dab-a6251fdcc30f" containerName="revoke-keys" Nov 01 22:59:33.438560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438419 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="63056725-77e3-4c7b-8132-1ec6325a1081" containerName="istio-proxy" Nov 01 22:59:33.438560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438427 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" Nov 01 22:59:33.438560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438436 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a8bacc-1268-463d-b491-b9a705807834" containerName="istio-proxy" Nov 01 22:59:33.438560 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.438444 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb05797-6bbf-45c1-a341-088aeadb04e0" containerName="payload-pre-processing" Nov 01 22:59:33.442718 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.442699 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.453392 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.453362 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp"] Nov 01 22:59:33.453716 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.453695 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-788d7b-openshift-default-dockercfg-8d6wp\"" Nov 01 22:59:33.518782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.518740 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.518945 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.518897 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-certs\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.519019 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.518948 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-data\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.519068 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.519045 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00290935-a1cc-4b90-b259-de633a083de1-istiod-ca-cert\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.519111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.519099 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00290935-a1cc-4b90-b259-de633a083de1-istio-podinfo\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.519189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.519177 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hcw5\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-kube-api-access-2hcw5\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.519234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.519213 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-istio-token\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.519314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.519302 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-credential-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.519375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.519364 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-envoy\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.620872 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.620820 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-credential-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.620915 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-envoy\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.620991 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621046 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-certs\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621094 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621077 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-data\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621134 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00290935-a1cc-4b90-b259-de633a083de1-istiod-ca-cert\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621194 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00290935-a1cc-4b90-b259-de633a083de1-istio-podinfo\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621249 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hcw5\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-kube-api-access-2hcw5\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621302 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-istio-token\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621355 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621324 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-credential-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621385 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-socket\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621713 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621441 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-certs\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621813 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621789 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-data\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.621976 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.621958 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00290935-a1cc-4b90-b259-de633a083de1-istiod-ca-cert\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.623410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.623383 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-envoy\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.624138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.624114 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00290935-a1cc-4b90-b259-de633a083de1-istio-podinfo\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.635372 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.635315 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-istio-token\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.639117 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.639091 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hcw5\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-kube-api-access-2hcw5\") pod \"e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:33.683181 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.683151 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:33.770405 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:33.770217 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:34.087701 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:59:34.087669 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00290935_a1cc_4b90_b259_de633a083de1.slice/crio-219e553f127f90df07660072d89b5124c686c0ef288c27306dfb80243987f909 WatchSource:0}: Error finding container 219e553f127f90df07660072d89b5124c686c0ef288c27306dfb80243987f909: Status 404 returned error can't find the container with id 219e553f127f90df07660072d89b5124c686c0ef288c27306dfb80243987f909 Nov 01 22:59:34.088628 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.088559 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp"] Nov 01 22:59:34.091425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.091308 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:59:34.091549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.091464 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:59:34.091549 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.091500 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 22:59:34.352249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.352164 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" event={"ID":"00290935-a1cc-4b90-b259-de633a083de1","Type":"ContainerStarted","Data":"e7be2de8a8ec2c64cb8f3dbaad546b12cbeb946a1b2affd1cb538bd9ba3dc4e8"} Nov 01 22:59:34.352249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.352228 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" event={"ID":"00290935-a1cc-4b90-b259-de633a083de1","Type":"ContainerStarted","Data":"219e553f127f90df07660072d89b5124c686c0ef288c27306dfb80243987f909"} Nov 01 22:59:34.374201 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.374142 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" podStartSLOduration=1.37412443 podStartE2EDuration="1.37412443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:34.370056975 +0000 UTC m=+2139.797934907" watchObservedRunningTime="2026-11-01 22:59:34.37412443 +0000 UTC m=+2139.802002361" Nov 01 22:59:34.555314 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.555276 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:34.771461 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:34.771428 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:35.501763 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:35.501716 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:35.772978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:35.772896 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 01 22:59:35.772978 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:35.772962 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 01 22:59:36.327606 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:36.327553 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:36.771988 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:36.771957 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 01 22:59:36.772375 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:36.772017 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 01 22:59:37.049765 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:37.049680 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:37.769916 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:37.769880 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:37.771586 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:37.771545 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 01 22:59:37.771744 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:37.771606 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 01 22:59:38.465640 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:38.465606 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:38.772233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:38.772145 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 01 22:59:38.772233 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:38.772207 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 01 22:59:39.152825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:39.152737 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:39.772059 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:39.772026 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 01 22:59:39.772469 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:39.772083 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 01 22:59:39.995262 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:39.995206 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:40.663226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:40.663190 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:40.865421 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:40.865375 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:40.866219 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:40.866193 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:40.867755 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:40.867732 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 22:59:41.029891 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:41.029808 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:41.561443 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:41.561413 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:42.307146 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.307112 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb"] Nov 01 22:59:42.313727 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.313537 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.319725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.319698 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:59:42.320747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.320726 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:59:42.320951 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.320738 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb"] Nov 01 22:59:42.356309 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.356278 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.356497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.356394 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.356565 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.356541 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzkw5\" (UniqueName: \"kubernetes.io/projected/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-kube-api-access-dzkw5\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.358897 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.358870 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:42.477622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.471810 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.477622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.471902 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.477622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.471979 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzkw5\" (UniqueName: \"kubernetes.io/projected/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-kube-api-access-dzkw5\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.477622 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.473430 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.491276 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.491203 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.492944 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.492918 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzkw5\" (UniqueName: \"kubernetes.io/projected/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-kube-api-access-dzkw5\") pod \"maas-api-revoke-sub-46ffd568-k9rzb\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.579049 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.578977 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d"] Nov 01 22:59:42.584391 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.584369 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.591782 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.591742 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d"] Nov 01 22:59:42.639881 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.639849 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:42.794842 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.794803 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.795039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.794896 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.795039 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.794993 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcbtc\" (UniqueName: \"kubernetes.io/projected/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-kube-api-access-hcbtc\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.896902 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.896814 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.897062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.896907 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.897062 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.896968 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcbtc\" (UniqueName: \"kubernetes.io/projected/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-kube-api-access-hcbtc\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.897970 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.897944 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.899416 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.899397 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.904825 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.904803 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcbtc\" (UniqueName: \"kubernetes.io/projected/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-kube-api-access-hcbtc\") pod \"maas-api-revoke-sub-8ecfa7c1-6tm9d\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:42.908943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:42.908919 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:43.025140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.025111 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb"] Nov 01 22:59:43.056856 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.056760 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:43.198468 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:59:43.198441 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4dbbb9_0a35_44f8_a1af_91e065b368b5.slice/crio-7eadc033498cabd39747a0be4de8cae36b2ccce4b53de3df603ea13cd70c9c32 WatchSource:0}: Error finding container 7eadc033498cabd39747a0be4de8cae36b2ccce4b53de3df603ea13cd70c9c32: Status 404 returned error can't find the container with id 7eadc033498cabd39747a0be4de8cae36b2ccce4b53de3df603ea13cd70c9c32 Nov 01 22:59:43.199743 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.199721 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d"] Nov 01 22:59:43.500849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.500814 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" event={"ID":"9dad7027-ccd0-46ad-b7d1-2e7236ec096a","Type":"ContainerStarted","Data":"017066b741a6b9800fcc025b61aefa74360bf7682d02c0921b0b12e51d2fc332"} Nov 01 22:59:43.501300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.500876 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" event={"ID":"9dad7027-ccd0-46ad-b7d1-2e7236ec096a","Type":"ContainerStarted","Data":"cf14f621803325a52d3540f8482e595391e53b11928c1df595146b08e2da35eb"} Nov 01 22:59:43.502749 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.502720 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" event={"ID":"0c4dbbb9-0a35-44f8-a1af-91e065b368b5","Type":"ContainerStarted","Data":"ef50c3e89bd1a37c16fd38e26c287ac87a735a4c65b4bb1898a4fc83c6bc16ca"} Nov 01 22:59:43.502875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.502777 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" event={"ID":"0c4dbbb9-0a35-44f8-a1af-91e065b368b5","Type":"ContainerStarted","Data":"7eadc033498cabd39747a0be4de8cae36b2ccce4b53de3df603ea13cd70c9c32"} Nov 01 22:59:43.518627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.518571 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" podStartSLOduration=1.518558465 podStartE2EDuration="1.518558465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:43.515563388 +0000 UTC m=+2148.943441345" watchObservedRunningTime="2026-11-01 22:59:43.518558465 +0000 UTC m=+2148.946436401" Nov 01 22:59:43.535627 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.535564 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" podStartSLOduration=1.535552691 podStartE2EDuration="1.535552691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:43.53091232 +0000 UTC m=+2148.958790251" watchObservedRunningTime="2026-11-01 22:59:43.535552691 +0000 UTC m=+2148.963430622" Nov 01 22:59:43.753300 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:43.753220 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:44.367519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:44.367433 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:44.513330 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:44.513296 4981 generic.go:356] "Generic (PLEG): container finished" podID="9dad7027-ccd0-46ad-b7d1-2e7236ec096a" containerID="017066b741a6b9800fcc025b61aefa74360bf7682d02c0921b0b12e51d2fc332" exitCode=0 Nov 01 22:59:44.513800 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:44.513336 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" event={"ID":"9dad7027-ccd0-46ad-b7d1-2e7236ec096a","Type":"ContainerDied","Data":"017066b741a6b9800fcc025b61aefa74360bf7682d02c0921b0b12e51d2fc332"} Nov 01 22:59:44.515543 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:44.515515 4981 generic.go:356] "Generic (PLEG): container finished" podID="0c4dbbb9-0a35-44f8-a1af-91e065b368b5" containerID="ef50c3e89bd1a37c16fd38e26c287ac87a735a4c65b4bb1898a4fc83c6bc16ca" exitCode=0 Nov 01 22:59:44.515747 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:44.515609 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" event={"ID":"0c4dbbb9-0a35-44f8-a1af-91e065b368b5","Type":"ContainerDied","Data":"ef50c3e89bd1a37c16fd38e26c287ac87a735a4c65b4bb1898a4fc83c6bc16ca"} Nov 01 22:59:44.527103 ip-10-0-138-87 kubenswrapper[4981]: E1101 22:59:44.527075 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 22:59:45.345602 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.337555 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:45.749106 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.749079 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:45.782742 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.782706 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:45.788746 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.788716 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-xks-ca\") pod \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " Nov 01 22:59:45.788882 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.788842 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-openshift-ca\") pod \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " Nov 01 22:59:45.789048 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.789029 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-kube-api-access-dzkw5\" (UniqueName: \"kubernetes.io/projected/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-kube-api-access-dzkw5\") pod \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\" (UID: \"9dad7027-ccd0-46ad-b7d1-2e7236ec096a\") " Nov 01 22:59:45.789210 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.789189 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-openshift-ca" pod "9dad7027-ccd0-46ad-b7d1-2e7236ec096a" (UID: "9dad7027-ccd0-46ad-b7d1-2e7236ec096a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:45.790140 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.790112 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:45.791081 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.791054 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-xks-ca" pod "9dad7027-ccd0-46ad-b7d1-2e7236ec096a" (UID: "9dad7027-ccd0-46ad-b7d1-2e7236ec096a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:59:45.791234 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.791217 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-kube-api-access-dzkw5" pod "9dad7027-ccd0-46ad-b7d1-2e7236ec096a" (UID: "9dad7027-ccd0-46ad-b7d1-2e7236ec096a"). InnerVolumeSpecName "kube-api-access-dzkw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:45.890973 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.890898 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-kube-api-access-hcbtc\" (UniqueName: \"kubernetes.io/projected/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-kube-api-access-hcbtc\") pod \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " Nov 01 22:59:45.891148 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.891029 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-xks-ca\") pod \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " Nov 01 22:59:45.891217 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.891175 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-openshift-ca\") pod \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\" (UID: \"0c4dbbb9-0a35-44f8-a1af-91e065b368b5\") " Nov 01 22:59:45.891562 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.891534 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-openshift-ca" pod "0c4dbbb9-0a35-44f8-a1af-91e065b368b5" (UID: "0c4dbbb9-0a35-44f8-a1af-91e065b368b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:45.891864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.891843 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:45.891864 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.891864 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzkw5\" (UniqueName: \"kubernetes.io/projected/9dad7027-ccd0-46ad-b7d1-2e7236ec096a-kube-api-access-dzkw5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:45.892012 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.891874 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:45.893099 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.893079 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-xks-ca" pod "0c4dbbb9-0a35-44f8-a1af-91e065b368b5" (UID: "0c4dbbb9-0a35-44f8-a1af-91e065b368b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:59:45.893188 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.893109 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-kube-api-access-hcbtc" pod "0c4dbbb9-0a35-44f8-a1af-91e065b368b5" (UID: "0c4dbbb9-0a35-44f8-a1af-91e065b368b5"). InnerVolumeSpecName "kube-api-access-hcbtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:45.974264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.974224 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:45.993272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.993243 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcbtc\" (UniqueName: \"kubernetes.io/projected/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-kube-api-access-hcbtc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:45.993272 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:45.993274 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c4dbbb9-0a35-44f8-a1af-91e065b368b5-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:46.534055 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:46.534027 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" Nov 01 22:59:46.534264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:46.534062 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d" event={"ID":"0c4dbbb9-0a35-44f8-a1af-91e065b368b5","Type":"ContainerDied","Data":"7eadc033498cabd39747a0be4de8cae36b2ccce4b53de3df603ea13cd70c9c32"} Nov 01 22:59:46.534264 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:46.534110 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eadc033498cabd39747a0be4de8cae36b2ccce4b53de3df603ea13cd70c9c32" Nov 01 22:59:46.535878 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:46.535852 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" event={"ID":"9dad7027-ccd0-46ad-b7d1-2e7236ec096a","Type":"ContainerDied","Data":"cf14f621803325a52d3540f8482e595391e53b11928c1df595146b08e2da35eb"} Nov 01 22:59:46.536021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:46.535898 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf14f621803325a52d3540f8482e595391e53b11928c1df595146b08e2da35eb" Nov 01 22:59:46.536021 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:46.535861 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb" Nov 01 22:59:46.909943 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:46.909861 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:47.625302 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:47.625261 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:48.298853 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:48.298817 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:48.963226 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:48.963193 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:49.643111 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:49.643080 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:50.373512 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:50.373474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:51.053953 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:51.053913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:51.739199 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:51.739157 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:52.537969 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:52.537931 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:53.339138 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:53.339107 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:53.978379 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:53.978347 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:54.669546 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:54.669515 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:55.037001 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.036960 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:55.586088 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.586052 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9"] Nov 01 22:59:55.586931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.586913 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9dad7027-ccd0-46ad-b7d1-2e7236ec096a" containerName="revoke-keys" Nov 01 22:59:55.586931 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.586931 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dad7027-ccd0-46ad-b7d1-2e7236ec096a" containerName="revoke-keys" Nov 01 22:59:55.587043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.586995 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0c4dbbb9-0a35-44f8-a1af-91e065b368b5" containerName="revoke-keys" Nov 01 22:59:55.587043 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.587000 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4dbbb9-0a35-44f8-a1af-91e065b368b5" containerName="revoke-keys" Nov 01 22:59:55.587127 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.587117 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4dbbb9-0a35-44f8-a1af-91e065b368b5" containerName="revoke-keys" Nov 01 22:59:55.587169 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.587131 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dad7027-ccd0-46ad-b7d1-2e7236ec096a" containerName="revoke-keys" Nov 01 22:59:55.591174 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.591158 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.594705 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.594681 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 22:59:55.595520 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.595491 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 22:59:55.598898 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.598873 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9"] Nov 01 22:59:55.633216 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.633186 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.633360 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.633309 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsv7r\" (UniqueName: \"kubernetes.io/projected/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-kube-api-access-vsv7r\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.633504 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.633483 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.673189 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.673162 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:55.738596 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.738539 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.739877 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.739845 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsv7r\" (UniqueName: \"kubernetes.io/projected/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-kube-api-access-vsv7r\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.740666 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.740639 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.748305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.739680 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.754678 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.754618 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsv7r\" (UniqueName: \"kubernetes.io/projected/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-kube-api-access-vsv7r\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.755875 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.755835 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f04fd0b1-fd6p9\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:55.914254 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:55.914174 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:56.057538 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:56.057508 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:56.154410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:56.154357 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:56.198829 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:59:56.198797 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cac79bc_72cd_4f75_9fb6_5c15879bdfef.slice/crio-c6192644e10928107fa5bdb5d0c1af431ef2d42526e5dc3c45b7bff652809ae0 WatchSource:0}: Error finding container c6192644e10928107fa5bdb5d0c1af431ef2d42526e5dc3c45b7bff652809ae0: Status 404 returned error can't find the container with id c6192644e10928107fa5bdb5d0c1af431ef2d42526e5dc3c45b7bff652809ae0 Nov 01 22:59:56.198989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:56.198912 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9"] Nov 01 22:59:56.619305 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:56.619273 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:56.659410 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:56.659374 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" event={"ID":"5cac79bc-72cd-4f75-9fb6-5c15879bdfef","Type":"ContainerStarted","Data":"068218a54bcbeb75ee704ab8d3e2c0afc984d9972fcc5d430bdce8692062ae87"} Nov 01 22:59:56.659664 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:56.659435 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" event={"ID":"5cac79bc-72cd-4f75-9fb6-5c15879bdfef","Type":"ContainerStarted","Data":"c6192644e10928107fa5bdb5d0c1af431ef2d42526e5dc3c45b7bff652809ae0"} Nov 01 22:59:56.689849 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:56.689789 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" podStartSLOduration=1.689774771 podStartE2EDuration="1.689774771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:56.688899237 +0000 UTC m=+2162.116777205" watchObservedRunningTime="2026-11-01 22:59:56.689774771 +0000 UTC m=+2162.117652702" Nov 01 22:59:57.354420 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:57.354375 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:57.671798 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:57.671718 4981 generic.go:356] "Generic (PLEG): container finished" podID="5cac79bc-72cd-4f75-9fb6-5c15879bdfef" containerID="068218a54bcbeb75ee704ab8d3e2c0afc984d9972fcc5d430bdce8692062ae87" exitCode=0 Nov 01 22:59:57.671961 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:57.671800 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" event={"ID":"5cac79bc-72cd-4f75-9fb6-5c15879bdfef","Type":"ContainerDied","Data":"068218a54bcbeb75ee704ab8d3e2c0afc984d9972fcc5d430bdce8692062ae87"} Nov 01 22:59:57.803663 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:57.803630 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:58.283425 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.283395 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254"] Nov 01 22:59:58.288900 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.288880 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.299733 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.298894 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254"] Nov 01 22:59:58.330321 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.330289 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:58.414600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.407048 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.414600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.407109 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqs6t\" (UniqueName: \"kubernetes.io/projected/03fff2db-3e5f-42e6-b407-ded0679a0c1c-kube-api-access-rqs6t\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.414600 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.407292 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.508938 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.508906 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.509108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.508947 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqs6t\" (UniqueName: \"kubernetes.io/projected/03fff2db-3e5f-42e6-b407-ded0679a0c1c-kube-api-access-rqs6t\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.509108 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.509097 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.509771 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.509744 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.511249 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.511230 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.517557 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.517527 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqs6t\" (UniqueName: \"kubernetes.io/projected/03fff2db-3e5f-42e6-b407-ded0679a0c1c-kube-api-access-rqs6t\") pod \"maas-api-revoke-sub-fcdd9d5c-q9254\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.621070 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.620993 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 22:59:58.886514 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.886489 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:58.909948 ip-10-0-138-87 kubenswrapper[4981]: W1101 22:59:58.909920 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fff2db_3e5f_42e6_b407_ded0679a0c1c.slice/crio-e4fadb1522058ced9cc072a4555ea169dd1561928c2e1488a5973b645831b5c8 WatchSource:0}: Error finding container e4fadb1522058ced9cc072a4555ea169dd1561928c2e1488a5973b645831b5c8: Status 404 returned error can't find the container with id e4fadb1522058ced9cc072a4555ea169dd1561928c2e1488a5973b645831b5c8 Nov 01 22:59:58.911202 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:58.911170 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254"] Nov 01 22:59:59.017519 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.017491 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-kube-api-access-vsv7r\" (UniqueName: \"kubernetes.io/projected/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-kube-api-access-vsv7r\") pod \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " Nov 01 22:59:59.017725 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.017646 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-xks-ca\") pod \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " Nov 01 22:59:59.017770 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.017741 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-openshift-ca\") pod \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\" (UID: \"5cac79bc-72cd-4f75-9fb6-5c15879bdfef\") " Nov 01 22:59:59.018131 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.018106 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-openshift-ca" pod "5cac79bc-72cd-4f75-9fb6-5c15879bdfef" (UID: "5cac79bc-72cd-4f75-9fb6-5c15879bdfef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 22:59:59.018438 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.018422 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:59.019879 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.019757 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-xks-ca" pod "5cac79bc-72cd-4f75-9fb6-5c15879bdfef" (UID: "5cac79bc-72cd-4f75-9fb6-5c15879bdfef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 22:59:59.020075 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.020049 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-kube-api-access-vsv7r" pod "5cac79bc-72cd-4f75-9fb6-5c15879bdfef" (UID: "5cac79bc-72cd-4f75-9fb6-5c15879bdfef"). InnerVolumeSpecName "kube-api-access-vsv7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 22:59:59.065551 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.065518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 22:59:59.139497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.125008 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsv7r\" (UniqueName: \"kubernetes.io/projected/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-kube-api-access-vsv7r\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:59.139497 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.125080 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cac79bc-72cd-4f75-9fb6-5c15879bdfef-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 22:59:59.693816 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.693786 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" Nov 01 22:59:59.694278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.693827 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9" event={"ID":"5cac79bc-72cd-4f75-9fb6-5c15879bdfef","Type":"ContainerDied","Data":"c6192644e10928107fa5bdb5d0c1af431ef2d42526e5dc3c45b7bff652809ae0"} Nov 01 22:59:59.694278 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.693883 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6192644e10928107fa5bdb5d0c1af431ef2d42526e5dc3c45b7bff652809ae0" Nov 01 22:59:59.695630 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.695603 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" event={"ID":"03fff2db-3e5f-42e6-b407-ded0679a0c1c","Type":"ContainerStarted","Data":"36710a7049e81c8fe1db5b83bc95e48f66275fe8a0a3485dba8c2088156bcbbd"} Nov 01 22:59:59.695989 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.695652 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" event={"ID":"03fff2db-3e5f-42e6-b407-ded0679a0c1c","Type":"ContainerStarted","Data":"e4fadb1522058ced9cc072a4555ea169dd1561928c2e1488a5973b645831b5c8"} Nov 01 22:59:59.722977 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.722935 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" podStartSLOduration=1.7229211549999999 podStartE2EDuration="1.722921155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:59.71559072 +0000 UTC m=+2165.143468647" watchObservedRunningTime="2026-11-01 22:59:59.722921155 +0000 UTC m=+2165.150799099" Nov 01 22:59:59.765110 ip-10-0-138-87 kubenswrapper[4981]: I1101 22:59:59.765080 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:00.151478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.151387 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n"] Nov 01 23:00:00.152468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.152441 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5cac79bc-72cd-4f75-9fb6-5c15879bdfef" containerName="revoke-keys" Nov 01 23:00:00.152468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.152467 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cac79bc-72cd-4f75-9fb6-5c15879bdfef" containerName="revoke-keys" Nov 01 23:00:00.152805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.152786 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cac79bc-72cd-4f75-9fb6-5c15879bdfef" containerName="revoke-keys" Nov 01 23:00:00.157269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.157244 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.164518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.164493 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj"] Nov 01 23:00:00.170239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.170215 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.170515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.170488 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988"] Nov 01 23:00:00.177872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.177611 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.178666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.178642 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b"] Nov 01 23:00:00.188275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.188249 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq"] Nov 01 23:00:00.189349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.189235 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.207277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.207196 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n"] Nov 01 23:00:00.207277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.207243 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj"] Nov 01 23:00:00.207277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.207252 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.210288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.210259 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988"] Nov 01 23:00:00.219725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.217756 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b"] Nov 01 23:00:00.228467 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.225533 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq"] Nov 01 23:00:00.240926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.237847 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw"] Nov 01 23:00:00.245440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.245395 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g"] Nov 01 23:00:00.247153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.246443 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.252038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.252016 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.259544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.259517 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.260169 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.260064 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhctw\" (UniqueName: \"kubernetes.io/projected/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-kube-api-access-qhctw\") pod \"maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.278155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.278128 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g"] Nov 01 23:00:00.287080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.287050 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw"] Nov 01 23:00:00.348268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.348230 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9"] Nov 01 23:00:00.356094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.356047 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.357698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.357674 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft"] Nov 01 23:00:00.362845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362275 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43a9074-cf62-4550-9509-e51ead257c0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.362845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362397 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhctw\" (UniqueName: \"kubernetes.io/projected/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-kube-api-access-qhctw\") pod \"maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.362845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362465 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jbd\" (UniqueName: \"kubernetes.io/projected/a43a9074-cf62-4550-9509-e51ead257c0a-kube-api-access-l8jbd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.362845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362518 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.362845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362681 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76m7\" (UniqueName: \"kubernetes.io/projected/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-kube-api-access-j76m7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.362845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362734 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.362845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362763 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0350611d-dea7-4f33-88a7-04f0fb719138-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.363309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362879 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k796q\" (UniqueName: \"kubernetes.io/projected/f2dad26e-4750-4dcc-b884-93aabfdcd334-kube-api-access-k796q\") pod \"maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.363309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.362906 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqt4t\" (UniqueName: \"kubernetes.io/projected/28bdfd58-37b7-479b-9633-7ed9e5b288c5-kube-api-access-nqt4t\") pod \"maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.363309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.363195 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw57v\" (UniqueName: \"kubernetes.io/projected/0350611d-dea7-4f33-88a7-04f0fb719138-kube-api-access-gw57v\") pod \"maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.363309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.363236 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bdfd58-37b7-479b-9633-7ed9e5b288c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.363309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.363271 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.363309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.363298 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dad26e-4750-4dcc-b884-93aabfdcd334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.363708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.363333 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dv6\" (UniqueName: \"kubernetes.io/projected/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-kube-api-access-s6dv6\") pod \"maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.364482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.364453 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.367823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.367799 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9"] Nov 01 23:00:00.367964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.367892 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.385788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.385757 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft"] Nov 01 23:00:00.391333 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.391305 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhctw\" (UniqueName: \"kubernetes.io/projected/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-kube-api-access-qhctw\") pod \"maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.393715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.393690 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw"] Nov 01 23:00:00.401537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.401433 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.408363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.408338 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f"] Nov 01 23:00:00.417022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.416974 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.418897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.418874 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw"] Nov 01 23:00:00.441950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.441879 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f"] Nov 01 23:00:00.447037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.446907 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79"] Nov 01 23:00:00.459327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.459301 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj"] Nov 01 23:00:00.459503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.459381 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.465536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.465500 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0350611d-dea7-4f33-88a7-04f0fb719138-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.465813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.465791 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmmx\" (UniqueName: \"kubernetes.io/projected/090be1a3-6eab-40e2-a821-f84b54f74ff8-kube-api-access-jvmmx\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.465986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.465971 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k796q\" (UniqueName: \"kubernetes.io/projected/f2dad26e-4750-4dcc-b884-93aabfdcd334-kube-api-access-k796q\") pod \"maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.466086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.466072 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqt4t\" (UniqueName: \"kubernetes.io/projected/28bdfd58-37b7-479b-9633-7ed9e5b288c5-kube-api-access-nqt4t\") pod \"maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.466342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.466321 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0350611d-dea7-4f33-88a7-04f0fb719138-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.466518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.466350 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj"] Nov 01 23:00:00.466518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.466469 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.466668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.466651 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw57v\" (UniqueName: \"kubernetes.io/projected/0350611d-dea7-4f33-88a7-04f0fb719138-kube-api-access-gw57v\") pod \"maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.466762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.466752 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bdfd58-37b7-479b-9633-7ed9e5b288c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.467393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.467378 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dad26e-4750-4dcc-b884-93aabfdcd334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.468022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468002 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dv6\" (UniqueName: \"kubernetes.io/projected/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-kube-api-access-s6dv6\") pod \"maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.468125 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468054 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/090be1a3-6eab-40e2-a821-f84b54f74ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.468125 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468105 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43a9074-cf62-4550-9509-e51ead257c0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.468233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468208 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jbd\" (UniqueName: \"kubernetes.io/projected/a43a9074-cf62-4550-9509-e51ead257c0a-kube-api-access-l8jbd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.468289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468235 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpbp\" (UniqueName: \"kubernetes.io/projected/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-kube-api-access-8kpbp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.468289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468262 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.468360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468293 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.468360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468350 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76m7\" (UniqueName: \"kubernetes.io/projected/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-kube-api-access-j76m7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.468425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468387 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.468552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.468530 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dad26e-4750-4dcc-b884-93aabfdcd334-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.468952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.467338 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bdfd58-37b7-479b-9633-7ed9e5b288c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.469332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.469309 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.470652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.470124 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.470886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.470842 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43a9074-cf62-4550-9509-e51ead257c0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.477736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.477706 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:00.481050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.481024 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79"] Nov 01 23:00:00.487160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.487106 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqt4t\" (UniqueName: \"kubernetes.io/projected/28bdfd58-37b7-479b-9633-7ed9e5b288c5-kube-api-access-nqt4t\") pod \"maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.489401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.489367 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dv6\" (UniqueName: \"kubernetes.io/projected/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-kube-api-access-s6dv6\") pod \"maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.493189 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.491153 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k796q\" (UniqueName: \"kubernetes.io/projected/f2dad26e-4750-4dcc-b884-93aabfdcd334-kube-api-access-k796q\") pod \"maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.500509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.500335 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw57v\" (UniqueName: \"kubernetes.io/projected/0350611d-dea7-4f33-88a7-04f0fb719138-kube-api-access-gw57v\") pod \"maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.500509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.500420 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76m7\" (UniqueName: \"kubernetes.io/projected/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-kube-api-access-j76m7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.502466 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.502439 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jbd\" (UniqueName: \"kubernetes.io/projected/a43a9074-cf62-4550-9509-e51ead257c0a-kube-api-access-l8jbd\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.503316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.503300 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:00.525658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.525554 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:00.536021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.535995 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:00.558192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.558154 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:00.574171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.574137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmmx\" (UniqueName: \"kubernetes.io/projected/090be1a3-6eab-40e2-a821-f84b54f74ff8-kube-api-access-jvmmx\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.574855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.574825 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6nwd\" (UniqueName: \"kubernetes.io/projected/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-kube-api-access-n6nwd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.575177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.575156 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e66cad-b147-4b3d-96eb-6b73e5a5144f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.575379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.575361 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt775\" (UniqueName: \"kubernetes.io/projected/1cf9a168-f16d-4282-8933-85e42c4893c8-kube-api-access-xt775\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.575605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.575565 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/090be1a3-6eab-40e2-a821-f84b54f74ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.576139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.576117 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf9a168-f16d-4282-8933-85e42c4893c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.576313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.576295 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2tng\" (UniqueName: \"kubernetes.io/projected/11e66cad-b147-4b3d-96eb-6b73e5a5144f-kube-api-access-m2tng\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.576531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.576511 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9f5f\" (UniqueName: \"kubernetes.io/projected/15a48509-ebff-4841-bea9-eead8afd4a82-kube-api-access-p9f5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.576844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.576824 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpbp\" (UniqueName: \"kubernetes.io/projected/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-kube-api-access-8kpbp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.577059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.577040 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.577212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.577194 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a48509-ebff-4841-bea9-eead8afd4a82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.577497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.577477 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.578937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.578918 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/090be1a3-6eab-40e2-a821-f84b54f74ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.580318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.580297 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.585507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.584250 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:00.601800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.596366 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpbp\" (UniqueName: \"kubernetes.io/projected/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-kube-api-access-8kpbp\") pod \"maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.617985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.617033 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmmx\" (UniqueName: \"kubernetes.io/projected/090be1a3-6eab-40e2-a821-f84b54f74ff8-kube-api-access-jvmmx\") pod \"maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.633998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.630623 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:00.679241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679205 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.679353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679298 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6nwd\" (UniqueName: \"kubernetes.io/projected/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-kube-api-access-n6nwd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.679434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679416 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e66cad-b147-4b3d-96eb-6b73e5a5144f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.679516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679500 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt775\" (UniqueName: \"kubernetes.io/projected/1cf9a168-f16d-4282-8933-85e42c4893c8-kube-api-access-xt775\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.679634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679618 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf9a168-f16d-4282-8933-85e42c4893c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.679703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679655 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2tng\" (UniqueName: \"kubernetes.io/projected/11e66cad-b147-4b3d-96eb-6b73e5a5144f-kube-api-access-m2tng\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.679754 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679710 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9f5f\" (UniqueName: \"kubernetes.io/projected/15a48509-ebff-4841-bea9-eead8afd4a82-kube-api-access-p9f5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.679854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.679839 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a48509-ebff-4841-bea9-eead8afd4a82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.686616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.680992 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.686774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.681976 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e66cad-b147-4b3d-96eb-6b73e5a5144f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.686774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.682914 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf9a168-f16d-4282-8933-85e42c4893c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.686774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.683012 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a48509-ebff-4841-bea9-eead8afd4a82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.688405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.688375 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:00.689157 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.689126 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt775\" (UniqueName: \"kubernetes.io/projected/1cf9a168-f16d-4282-8933-85e42c4893c8-kube-api-access-xt775\") pod \"maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.690154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.690121 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2tng\" (UniqueName: \"kubernetes.io/projected/11e66cad-b147-4b3d-96eb-6b73e5a5144f-kube-api-access-m2tng\") pod \"maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.691147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.691122 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9f5f\" (UniqueName: \"kubernetes.io/projected/15a48509-ebff-4841-bea9-eead8afd4a82-kube-api-access-p9f5f\") pod \"maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.699981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.697246 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6nwd\" (UniqueName: \"kubernetes.io/projected/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-kube-api-access-n6nwd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.752480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.751799 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:00.760880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.760851 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:00.796083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.795284 4981 generic.go:356] "Generic (PLEG): container finished" podID="03fff2db-3e5f-42e6-b407-ded0679a0c1c" containerID="36710a7049e81c8fe1db5b83bc95e48f66275fe8a0a3485dba8c2088156bcbbd" exitCode=0 Nov 01 23:00:00.805372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.805305 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:00.810064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.810014 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" event={"ID":"03fff2db-3e5f-42e6-b407-ded0679a0c1c","Type":"ContainerDied","Data":"36710a7049e81c8fe1db5b83bc95e48f66275fe8a0a3485dba8c2088156bcbbd"} Nov 01 23:00:00.960005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.959974 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:00.970227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.969948 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:00.974791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:00.974248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:01.016629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.016564 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n"] Nov 01 23:00:01.093731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.093696 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj"] Nov 01 23:00:01.119534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.119498 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988"] Nov 01 23:00:01.145145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.132602 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b"] Nov 01 23:00:01.367337 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:01.367292 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dad26e_4750_4dcc_b884_93aabfdcd334.slice/crio-e583be60db9cb288a8eeca49edf48ad254037e0075072e84b6c6fe6058053bd7 WatchSource:0}: Error finding container e583be60db9cb288a8eeca49edf48ad254037e0075072e84b6c6fe6058053bd7: Status 404 returned error can't find the container with id e583be60db9cb288a8eeca49edf48ad254037e0075072e84b6c6fe6058053bd7 Nov 01 23:00:01.368800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.368772 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq"] Nov 01 23:00:01.455809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.455774 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw"] Nov 01 23:00:01.459803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.459780 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g"] Nov 01 23:00:01.483901 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:01.483875 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae1171c_3cf2_43db_b6c1_2a1469875ee9.slice/crio-c11ae6a94be65005954ee3d0e7ef15d204d1c60162c7fb42e74af6374cd3f404 WatchSource:0}: Error finding container c11ae6a94be65005954ee3d0e7ef15d204d1c60162c7fb42e74af6374cd3f404: Status 404 returned error can't find the container with id c11ae6a94be65005954ee3d0e7ef15d204d1c60162c7fb42e74af6374cd3f404 Nov 01 23:00:01.515828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.515327 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft"] Nov 01 23:00:01.538407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.538381 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw"] Nov 01 23:00:01.540515 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:01.540492 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a48509_ebff_4841_bea9_eead8afd4a82.slice/crio-2f87fa355a1fc97456bdcbe28fd25bffc4e147ba0f4269c686be7c7da5d55ea2 WatchSource:0}: Error finding container 2f87fa355a1fc97456bdcbe28fd25bffc4e147ba0f4269c686be7c7da5d55ea2: Status 404 returned error can't find the container with id 2f87fa355a1fc97456bdcbe28fd25bffc4e147ba0f4269c686be7c7da5d55ea2 Nov 01 23:00:01.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.548900 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9"] Nov 01 23:00:01.550703 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:01.550662 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78d0d3e_6f2d_4986_8c07_eb9fb66f8b9b.slice/crio-42e5113e0de7d13fc374d754dbb4dd6d6bfac94a34c696a73d9df2f89a7c7b8d WatchSource:0}: Error finding container 42e5113e0de7d13fc374d754dbb4dd6d6bfac94a34c696a73d9df2f89a7c7b8d: Status 404 returned error can't find the container with id 42e5113e0de7d13fc374d754dbb4dd6d6bfac94a34c696a73d9df2f89a7c7b8d Nov 01 23:00:01.574750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.574727 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79"] Nov 01 23:00:01.586475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.586449 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f"] Nov 01 23:00:01.601300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.601276 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj"] Nov 01 23:00:01.678932 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:01.678857 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf9a168_f16d_4282_8933_85e42c4893c8.slice/crio-8f4ad9bad5316cc6b18c9ab7bc2d1feea7d91761f0a8482d2051365d4ce99c46 WatchSource:0}: Error finding container 8f4ad9bad5316cc6b18c9ab7bc2d1feea7d91761f0a8482d2051365d4ce99c46: Status 404 returned error can't find the container with id 8f4ad9bad5316cc6b18c9ab7bc2d1feea7d91761f0a8482d2051365d4ce99c46 Nov 01 23:00:01.739106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.739077 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:01.851934 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:01.851892 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e66cad_b147_4b3d_96eb_6b73e5a5144f.slice/crio-e239d12af67b63b11b8e9c26633220c39ec76cab7e40bcbd87492a449be341b1 WatchSource:0}: Error finding container e239d12af67b63b11b8e9c26633220c39ec76cab7e40bcbd87492a449be341b1: Status 404 returned error can't find the container with id e239d12af67b63b11b8e9c26633220c39ec76cab7e40bcbd87492a449be341b1 Nov 01 23:00:01.861458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.861423 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerStarted","Data":"c11ae6a94be65005954ee3d0e7ef15d204d1c60162c7fb42e74af6374cd3f404"} Nov 01 23:00:01.863040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.862984 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerStarted","Data":"2f87fa355a1fc97456bdcbe28fd25bffc4e147ba0f4269c686be7c7da5d55ea2"} Nov 01 23:00:01.864460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.864438 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" event={"ID":"0350611d-dea7-4f33-88a7-04f0fb719138","Type":"ContainerStarted","Data":"c4b9b219aada14cc2066b7af4fc5cab4e6c26ad38743f6752a8f601d0517e424"} Nov 01 23:00:01.866073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.866047 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerStarted","Data":"42e5113e0de7d13fc374d754dbb4dd6d6bfac94a34c696a73d9df2f89a7c7b8d"} Nov 01 23:00:01.867192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.867172 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" event={"ID":"f2dad26e-4750-4dcc-b884-93aabfdcd334","Type":"ContainerStarted","Data":"e583be60db9cb288a8eeca49edf48ad254037e0075072e84b6c6fe6058053bd7"} Nov 01 23:00:01.868547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.868525 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerStarted","Data":"d0103ce19dce7798ef5e432ee93407a46e5e1d6d93d9e8876de6d955f0c39e51"} Nov 01 23:00:01.870265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.870127 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerStarted","Data":"8f4ad9bad5316cc6b18c9ab7bc2d1feea7d91761f0a8482d2051365d4ce99c46"} Nov 01 23:00:01.871905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.871878 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" event={"ID":"090be1a3-6eab-40e2-a821-f84b54f74ff8","Type":"ContainerStarted","Data":"ee4bd05cb9d77f145770bc5693cb337b6d1c4c0963c6368807e8a1ff79296bea"} Nov 01 23:00:01.873358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.873333 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" event={"ID":"28bdfd58-37b7-479b-9633-7ed9e5b288c5","Type":"ContainerStarted","Data":"0ba812459bda958839d5e003ad71fb8d9e766b99ce73963334ff0b2b495e3040"} Nov 01 23:00:01.875131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.875105 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" event={"ID":"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a","Type":"ContainerStarted","Data":"3eb8844e01d3ee16e070e3e9ba7d7bef8ed882651caa8a25ded412fa44e0eb0d"} Nov 01 23:00:01.879141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.879067 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" event={"ID":"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec","Type":"ContainerStarted","Data":"6b42424c6dea00d0fa0958695d8284aff129eecec72473229d67543e9a41b288"} Nov 01 23:00:01.881664 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:01.881605 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerStarted","Data":"3f70cb01fef05806489ddb298801a5441b9c98b53728ef6dd51a277af6c2ceed"} Nov 01 23:00:02.096627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.096596 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 23:00:02.136765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.136717 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-openshift-ca\") pod \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " Nov 01 23:00:02.136936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.136774 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fff2db-3e5f-42e6-b407-ded0679a0c1c-kube-api-access-rqs6t\" (UniqueName: \"kubernetes.io/projected/03fff2db-3e5f-42e6-b407-ded0679a0c1c-kube-api-access-rqs6t\") pod \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " Nov 01 23:00:02.137119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.137096 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-xks-ca\") pod \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\" (UID: \"03fff2db-3e5f-42e6-b407-ded0679a0c1c\") " Nov 01 23:00:02.137205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.137167 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-openshift-ca" pod "03fff2db-3e5f-42e6-b407-ded0679a0c1c" (UID: "03fff2db-3e5f-42e6-b407-ded0679a0c1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:02.138029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.137997 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:02.139560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.139531 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-xks-ca" pod "03fff2db-3e5f-42e6-b407-ded0679a0c1c" (UID: "03fff2db-3e5f-42e6-b407-ded0679a0c1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:02.139782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.139760 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fff2db-3e5f-42e6-b407-ded0679a0c1c-kube-api-access-rqs6t" pod "03fff2db-3e5f-42e6-b407-ded0679a0c1c" (UID: "03fff2db-3e5f-42e6-b407-ded0679a0c1c"). InnerVolumeSpecName "kube-api-access-rqs6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:02.239522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.239487 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03fff2db-3e5f-42e6-b407-ded0679a0c1c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:02.239522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.239521 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqs6t\" (UniqueName: \"kubernetes.io/projected/03fff2db-3e5f-42e6-b407-ded0679a0c1c-kube-api-access-rqs6t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:02.631071 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:02.631037 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bae0ed_a8fd_442f_aaf6_acd7aa5fbfec.slice/crio-8556fb8d2bd314fa4e68ecd4ffd5e058d4edd117e3b5c8e49a00c95d788e3000.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:00:02.631258 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:02.631105 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f29e1ad_6bae_4ab5_a21c_7c4505e9606a.slice/crio-conmon-50f931b92e815793e1e7f8d5988643c2f44057f329821e6c7d6d762a8f60b93b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bae0ed_a8fd_442f_aaf6_acd7aa5fbfec.slice/crio-conmon-8556fb8d2bd314fa4e68ecd4ffd5e058d4edd117e3b5c8e49a00c95d788e3000.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bae0ed_a8fd_442f_aaf6_acd7aa5fbfec.slice/crio-8556fb8d2bd314fa4e68ecd4ffd5e058d4edd117e3b5c8e49a00c95d788e3000.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f29e1ad_6bae_4ab5_a21c_7c4505e9606a.slice/crio-50f931b92e815793e1e7f8d5988643c2f44057f329821e6c7d6d762a8f60b93b.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:00:02.657785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.657748 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:02.893907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.893872 4981 generic.go:356] "Generic (PLEG): container finished" podID="0350611d-dea7-4f33-88a7-04f0fb719138" containerID="5ca42035dbd8fb1518a6c9d9c34763f4899bf821552d611658c7f79c83f212a6" exitCode=60 Nov 01 23:00:02.894345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.893951 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" event={"ID":"0350611d-dea7-4f33-88a7-04f0fb719138","Type":"ContainerDied","Data":"5ca42035dbd8fb1518a6c9d9c34763f4899bf821552d611658c7f79c83f212a6"} Nov 01 23:00:02.894345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.894332 4981 scope.go:122] "RemoveContainer" containerID="5ca42035dbd8fb1518a6c9d9c34763f4899bf821552d611658c7f79c83f212a6" Nov 01 23:00:02.895866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.895833 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerStarted","Data":"271b7f692b2dc05cecbdacb94162139e52c88e8a6c96b721d391ceab342634a5"} Nov 01 23:00:02.898702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.898677 4981 generic.go:356] "Generic (PLEG): container finished" podID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerID="ebec1d6fac13fc991886969fbb1f02ba82a6f978473dfadb0478b04e90da4ce5" exitCode=60 Nov 01 23:00:02.898850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.898806 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" event={"ID":"f2dad26e-4750-4dcc-b884-93aabfdcd334","Type":"ContainerDied","Data":"ebec1d6fac13fc991886969fbb1f02ba82a6f978473dfadb0478b04e90da4ce5"} Nov 01 23:00:02.899209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.899189 4981 scope.go:122] "RemoveContainer" containerID="ebec1d6fac13fc991886969fbb1f02ba82a6f978473dfadb0478b04e90da4ce5" Nov 01 23:00:02.903933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.903282 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerStarted","Data":"a03d0bab573e073d6155902852445fd8c6392af74f7e04c155021ccbe6f69fa8"} Nov 01 23:00:02.905244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.905219 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerStarted","Data":"75374f19c39da0b782defccf99e654bb5baa5ac2a1cd93a38788dc6630c21f59"} Nov 01 23:00:02.905352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.905264 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerStarted","Data":"e239d12af67b63b11b8e9c26633220c39ec76cab7e40bcbd87492a449be341b1"} Nov 01 23:00:02.906907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.906884 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerStarted","Data":"cca4451f2e878f9359696547375468114020c09020983399f0e2995d97b6c983"} Nov 01 23:00:02.909280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.909256 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" event={"ID":"090be1a3-6eab-40e2-a821-f84b54f74ff8","Type":"ContainerStarted","Data":"7171ee9d6f720403e4a6344a6d6fcb48d963e0b7f541c66eae7a058cf1e5bd72"} Nov 01 23:00:02.911510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.911487 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" event={"ID":"03fff2db-3e5f-42e6-b407-ded0679a0c1c","Type":"ContainerDied","Data":"e4fadb1522058ced9cc072a4555ea169dd1561928c2e1488a5973b645831b5c8"} Nov 01 23:00:02.911666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.911527 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4fadb1522058ced9cc072a4555ea169dd1561928c2e1488a5973b645831b5c8" Nov 01 23:00:02.911666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.911532 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254" Nov 01 23:00:02.915017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.914993 4981 generic.go:356] "Generic (PLEG): container finished" podID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerID="43c80411bdb0b9f600169dd301b8441949415ce83e4a0170bc4a7a639cc5641d" exitCode=60 Nov 01 23:00:02.915123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.915106 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" event={"ID":"28bdfd58-37b7-479b-9633-7ed9e5b288c5","Type":"ContainerDied","Data":"43c80411bdb0b9f600169dd301b8441949415ce83e4a0170bc4a7a639cc5641d"} Nov 01 23:00:02.915507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.915488 4981 scope.go:122] "RemoveContainer" containerID="43c80411bdb0b9f600169dd301b8441949415ce83e4a0170bc4a7a639cc5641d" Nov 01 23:00:02.918324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.918305 4981 generic.go:356] "Generic (PLEG): container finished" podID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerID="50f931b92e815793e1e7f8d5988643c2f44057f329821e6c7d6d762a8f60b93b" exitCode=60 Nov 01 23:00:02.918414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.918370 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" event={"ID":"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a","Type":"ContainerDied","Data":"50f931b92e815793e1e7f8d5988643c2f44057f329821e6c7d6d762a8f60b93b"} Nov 01 23:00:02.918746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.918729 4981 scope.go:122] "RemoveContainer" containerID="50f931b92e815793e1e7f8d5988643c2f44057f329821e6c7d6d762a8f60b93b" Nov 01 23:00:02.920187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.920170 4981 generic.go:356] "Generic (PLEG): container finished" podID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerID="8556fb8d2bd314fa4e68ecd4ffd5e058d4edd117e3b5c8e49a00c95d788e3000" exitCode=60 Nov 01 23:00:02.920239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.920223 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" event={"ID":"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec","Type":"ContainerDied","Data":"8556fb8d2bd314fa4e68ecd4ffd5e058d4edd117e3b5c8e49a00c95d788e3000"} Nov 01 23:00:02.920516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.920503 4981 scope.go:122] "RemoveContainer" containerID="8556fb8d2bd314fa4e68ecd4ffd5e058d4edd117e3b5c8e49a00c95d788e3000" Nov 01 23:00:02.923173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.923153 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerStarted","Data":"6f6083f6144d4666c5023faafd73237c26da83980ab2187366f922fe553a83c7"} Nov 01 23:00:02.925111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.925089 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerStarted","Data":"85134349c28c858786628bde6313d56e779664897a7b3172d9563efa8478d93a"} Nov 01 23:00:02.931201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.931179 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerStarted","Data":"cb89cce99eabfa67b5f659682f4462760681dc79351653ad6e188794fe522199"} Nov 01 23:00:02.937364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:02.937313 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" podStartSLOduration=2.937297305 podStartE2EDuration="2.937297305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.934570959 +0000 UTC m=+2168.362448882" watchObservedRunningTime="2026-11-01 23:00:02.937297305 +0000 UTC m=+2168.365175236" Nov 01 23:00:03.013043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.012969 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" podStartSLOduration=3.012950206 podStartE2EDuration="3.012950206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.006007146 +0000 UTC m=+2168.433885078" watchObservedRunningTime="2026-11-01 23:00:03.012950206 +0000 UTC m=+2168.440828138" Nov 01 23:00:03.029997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.029944 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" podStartSLOduration=3.029928469 podStartE2EDuration="3.029928469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.026957485 +0000 UTC m=+2168.454835417" watchObservedRunningTime="2026-11-01 23:00:03.029928469 +0000 UTC m=+2168.457806402" Nov 01 23:00:03.054814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.054747 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" podStartSLOduration=3.054727091 podStartE2EDuration="3.054727091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.048953443 +0000 UTC m=+2168.476831375" watchObservedRunningTime="2026-11-01 23:00:03.054727091 +0000 UTC m=+2168.482605023" Nov 01 23:00:03.070787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.070731 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" podStartSLOduration=3.070717859 podStartE2EDuration="3.070717859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.065017856 +0000 UTC m=+2168.492895789" watchObservedRunningTime="2026-11-01 23:00:03.070717859 +0000 UTC m=+2168.498595790" Nov 01 23:00:03.091637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.091570 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" podStartSLOduration=3.091554076 podStartE2EDuration="3.091554076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.086163282 +0000 UTC m=+2168.514041213" watchObservedRunningTime="2026-11-01 23:00:03.091554076 +0000 UTC m=+2168.519432057" Nov 01 23:00:03.106523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.106459 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" podStartSLOduration=3.106444626 podStartE2EDuration="3.106444626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.105300109 +0000 UTC m=+2168.533178031" watchObservedRunningTime="2026-11-01 23:00:03.106444626 +0000 UTC m=+2168.534322556" Nov 01 23:00:03.137812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.137757 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" podStartSLOduration=3.137744852 podStartE2EDuration="3.137744852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.136471654 +0000 UTC m=+2168.564349586" watchObservedRunningTime="2026-11-01 23:00:03.137744852 +0000 UTC m=+2168.565622776" Nov 01 23:00:03.467791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.467755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:03.945412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.945313 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" event={"ID":"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec","Type":"ContainerStarted","Data":"66eb1d03d5665ba0ecc48f6dcc1a7360e2079632e94a6105587a9bbbd9a5454c"} Nov 01 23:00:03.947104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.947080 4981 generic.go:356] "Generic (PLEG): container finished" podID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerID="6f6083f6144d4666c5023faafd73237c26da83980ab2187366f922fe553a83c7" exitCode=60 Nov 01 23:00:03.947249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.947218 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerDied","Data":"6f6083f6144d4666c5023faafd73237c26da83980ab2187366f922fe553a83c7"} Nov 01 23:00:03.947551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.947531 4981 scope.go:122] "RemoveContainer" containerID="6f6083f6144d4666c5023faafd73237c26da83980ab2187366f922fe553a83c7" Nov 01 23:00:03.949185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.949164 4981 generic.go:356] "Generic (PLEG): container finished" podID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerID="85134349c28c858786628bde6313d56e779664897a7b3172d9563efa8478d93a" exitCode=60 Nov 01 23:00:03.949304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.949250 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerDied","Data":"85134349c28c858786628bde6313d56e779664897a7b3172d9563efa8478d93a"} Nov 01 23:00:03.949503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.949484 4981 scope.go:122] "RemoveContainer" containerID="85134349c28c858786628bde6313d56e779664897a7b3172d9563efa8478d93a" Nov 01 23:00:03.952225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.952208 4981 generic.go:356] "Generic (PLEG): container finished" podID="15a48509-ebff-4841-bea9-eead8afd4a82" containerID="cb89cce99eabfa67b5f659682f4462760681dc79351653ad6e188794fe522199" exitCode=60 Nov 01 23:00:03.952293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.952234 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerDied","Data":"cb89cce99eabfa67b5f659682f4462760681dc79351653ad6e188794fe522199"} Nov 01 23:00:03.952520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.952503 4981 scope.go:122] "RemoveContainer" containerID="cb89cce99eabfa67b5f659682f4462760681dc79351653ad6e188794fe522199" Nov 01 23:00:03.955305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.955281 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" event={"ID":"0350611d-dea7-4f33-88a7-04f0fb719138","Type":"ContainerStarted","Data":"fad5cdae5685b74561184c77269d965f48089007a864b5429c48fb3af0b74e54"} Nov 01 23:00:03.957222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.957203 4981 generic.go:356] "Generic (PLEG): container finished" podID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerID="271b7f692b2dc05cecbdacb94162139e52c88e8a6c96b721d391ceab342634a5" exitCode=60 Nov 01 23:00:03.957324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.957282 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerDied","Data":"271b7f692b2dc05cecbdacb94162139e52c88e8a6c96b721d391ceab342634a5"} Nov 01 23:00:03.957500 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.957485 4981 scope.go:122] "RemoveContainer" containerID="271b7f692b2dc05cecbdacb94162139e52c88e8a6c96b721d391ceab342634a5" Nov 01 23:00:03.959553 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.959520 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" event={"ID":"f2dad26e-4750-4dcc-b884-93aabfdcd334","Type":"ContainerStarted","Data":"aee23b392f56e848678916c05dbcc9318e2b10c99709c888e641b47fc77d5621"} Nov 01 23:00:03.961341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.961319 4981 generic.go:356] "Generic (PLEG): container finished" podID="a43a9074-cf62-4550-9509-e51ead257c0a" containerID="a03d0bab573e073d6155902852445fd8c6392af74f7e04c155021ccbe6f69fa8" exitCode=60 Nov 01 23:00:03.961449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.961384 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerDied","Data":"a03d0bab573e073d6155902852445fd8c6392af74f7e04c155021ccbe6f69fa8"} Nov 01 23:00:03.961676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.961657 4981 scope.go:122] "RemoveContainer" containerID="a03d0bab573e073d6155902852445fd8c6392af74f7e04c155021ccbe6f69fa8" Nov 01 23:00:03.964645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.964306 4981 generic.go:356] "Generic (PLEG): container finished" podID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerID="75374f19c39da0b782defccf99e654bb5baa5ac2a1cd93a38788dc6630c21f59" exitCode=60 Nov 01 23:00:03.964645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.964390 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerDied","Data":"75374f19c39da0b782defccf99e654bb5baa5ac2a1cd93a38788dc6630c21f59"} Nov 01 23:00:03.964733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.964652 4981 scope.go:122] "RemoveContainer" containerID="75374f19c39da0b782defccf99e654bb5baa5ac2a1cd93a38788dc6630c21f59" Nov 01 23:00:03.966209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.966188 4981 generic.go:356] "Generic (PLEG): container finished" podID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerID="cca4451f2e878f9359696547375468114020c09020983399f0e2995d97b6c983" exitCode=60 Nov 01 23:00:03.966299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.966245 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerDied","Data":"cca4451f2e878f9359696547375468114020c09020983399f0e2995d97b6c983"} Nov 01 23:00:03.966479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.966465 4981 scope.go:122] "RemoveContainer" containerID="cca4451f2e878f9359696547375468114020c09020983399f0e2995d97b6c983" Nov 01 23:00:03.968289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.968269 4981 generic.go:356] "Generic (PLEG): container finished" podID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerID="7171ee9d6f720403e4a6344a6d6fcb48d963e0b7f541c66eae7a058cf1e5bd72" exitCode=60 Nov 01 23:00:03.968382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.968316 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" event={"ID":"090be1a3-6eab-40e2-a821-f84b54f74ff8","Type":"ContainerDied","Data":"7171ee9d6f720403e4a6344a6d6fcb48d963e0b7f541c66eae7a058cf1e5bd72"} Nov 01 23:00:03.968616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.968589 4981 scope.go:122] "RemoveContainer" containerID="7171ee9d6f720403e4a6344a6d6fcb48d963e0b7f541c66eae7a058cf1e5bd72" Nov 01 23:00:03.972969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.970665 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" event={"ID":"28bdfd58-37b7-479b-9633-7ed9e5b288c5","Type":"ContainerStarted","Data":"3b17429a23c01391aa0e11c3c10cb0cc8206cd27587eb9ac29ce71104ab187a8"} Nov 01 23:00:03.978094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.978035 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" podStartSLOduration=3.978017545 podStartE2EDuration="3.978017545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.971480302 +0000 UTC m=+2169.399358235" watchObservedRunningTime="2026-11-01 23:00:03.978017545 +0000 UTC m=+2169.405895478" Nov 01 23:00:03.981589 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:03.980759 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" event={"ID":"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a","Type":"ContainerStarted","Data":"1a2fb20055769418a98cbf3d5499c78e76e0daecdb57b080f3148863922c0ab9"} Nov 01 23:00:04.080848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.080799 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" podStartSLOduration=4.080785895 podStartE2EDuration="4.080785895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.078794278 +0000 UTC m=+2169.506672212" watchObservedRunningTime="2026-11-01 23:00:04.080785895 +0000 UTC m=+2169.508663826" Nov 01 23:00:04.141038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.140984 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" podStartSLOduration=4.140971175 podStartE2EDuration="4.140971175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.139080975 +0000 UTC m=+2169.566958907" watchObservedRunningTime="2026-11-01 23:00:04.140971175 +0000 UTC m=+2169.568849106" Nov 01 23:00:04.238758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.238657 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" podStartSLOduration=4.238642876 podStartE2EDuration="4.238642876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.232981826 +0000 UTC m=+2169.660859758" watchObservedRunningTime="2026-11-01 23:00:04.238642876 +0000 UTC m=+2169.666520807" Nov 01 23:00:04.254403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.254350 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" podStartSLOduration=4.254335862 podStartE2EDuration="4.254335862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.250727986 +0000 UTC m=+2169.678605920" watchObservedRunningTime="2026-11-01 23:00:04.254335862 +0000 UTC m=+2169.682213847" Nov 01 23:00:04.424396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.424366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:04.992943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.992907 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerStarted","Data":"1a7b9e11eaf745d1b022a628b6c5c9cec20569e0635fff3f108469f1cf7189b8"} Nov 01 23:00:04.995112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.995085 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerStarted","Data":"343e4073125bd391d60fdc8cfe7bbbec1eabe49161d27112f9ed1e21d505b174"} Nov 01 23:00:04.997077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.997053 4981 generic.go:356] "Generic (PLEG): container finished" podID="0350611d-dea7-4f33-88a7-04f0fb719138" containerID="fad5cdae5685b74561184c77269d965f48089007a864b5429c48fb3af0b74e54" exitCode=60 Nov 01 23:00:04.997238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.997135 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" event={"ID":"0350611d-dea7-4f33-88a7-04f0fb719138","Type":"ContainerDied","Data":"fad5cdae5685b74561184c77269d965f48089007a864b5429c48fb3af0b74e54"} Nov 01 23:00:04.997238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.997185 4981 scope.go:122] "RemoveContainer" containerID="5ca42035dbd8fb1518a6c9d9c34763f4899bf821552d611658c7f79c83f212a6" Nov 01 23:00:04.997476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.997357 4981 scope.go:122] "RemoveContainer" containerID="fad5cdae5685b74561184c77269d965f48089007a864b5429c48fb3af0b74e54" Nov 01 23:00:04.999756 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:04.998422 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988_odh-ai-gateway-infra(0350611d-dea7-4f33-88a7-04f0fb719138)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" Nov 01 23:00:04.999862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:04.999840 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerStarted","Data":"337724fa93aff26d135c036d3ddc42c122b15c38a3b0089cc65237c7e0c5e29f"} Nov 01 23:00:05.001858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.001829 4981 generic.go:356] "Generic (PLEG): container finished" podID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerID="aee23b392f56e848678916c05dbcc9318e2b10c99709c888e641b47fc77d5621" exitCode=60 Nov 01 23:00:05.001962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.001908 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" event={"ID":"f2dad26e-4750-4dcc-b884-93aabfdcd334","Type":"ContainerDied","Data":"aee23b392f56e848678916c05dbcc9318e2b10c99709c888e641b47fc77d5621"} Nov 01 23:00:05.002155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.002140 4981 scope.go:122] "RemoveContainer" containerID="aee23b392f56e848678916c05dbcc9318e2b10c99709c888e641b47fc77d5621" Nov 01 23:00:05.002474 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:05.002448 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq_odh-ai-gateway-infra(f2dad26e-4750-4dcc-b884-93aabfdcd334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" Nov 01 23:00:05.004418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.004393 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerStarted","Data":"14dd8960a7f754c04dc58fae816e96072ec45e3a3a9cba8462c5315b46a2eb69"} Nov 01 23:00:05.006681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.006653 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerStarted","Data":"fd65089478afbdfb5b314581081848cc7c377786d694f8211b74caf72b71b0fd"} Nov 01 23:00:05.008710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.008689 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerStarted","Data":"92576b20ee575e695975e4d25f2a891977c0f96a62b392bd678fdd01a7ba9c78"} Nov 01 23:00:05.010636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.010609 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" event={"ID":"090be1a3-6eab-40e2-a821-f84b54f74ff8","Type":"ContainerStarted","Data":"e345bd906cd0d089424ed92c02403a6ea6206bbad8f5079bf892e19c919decbd"} Nov 01 23:00:05.017731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.017706 4981 scope.go:122] "RemoveContainer" containerID="ebec1d6fac13fc991886969fbb1f02ba82a6f978473dfadb0478b04e90da4ce5" Nov 01 23:00:05.017850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.017796 4981 generic.go:356] "Generic (PLEG): container finished" podID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerID="3b17429a23c01391aa0e11c3c10cb0cc8206cd27587eb9ac29ce71104ab187a8" exitCode=60 Nov 01 23:00:05.017938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.017906 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" event={"ID":"28bdfd58-37b7-479b-9633-7ed9e5b288c5","Type":"ContainerDied","Data":"3b17429a23c01391aa0e11c3c10cb0cc8206cd27587eb9ac29ce71104ab187a8"} Nov 01 23:00:05.018208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.018191 4981 scope.go:122] "RemoveContainer" containerID="3b17429a23c01391aa0e11c3c10cb0cc8206cd27587eb9ac29ce71104ab187a8" Nov 01 23:00:05.018653 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:05.018616 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b_odh-ai-gateway-infra(28bdfd58-37b7-479b-9633-7ed9e5b288c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" Nov 01 23:00:05.020028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.020007 4981 generic.go:356] "Generic (PLEG): container finished" podID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerID="1a2fb20055769418a98cbf3d5499c78e76e0daecdb57b080f3148863922c0ab9" exitCode=60 Nov 01 23:00:05.020160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.020079 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" event={"ID":"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a","Type":"ContainerDied","Data":"1a2fb20055769418a98cbf3d5499c78e76e0daecdb57b080f3148863922c0ab9"} Nov 01 23:00:05.020349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.020332 4981 scope.go:122] "RemoveContainer" containerID="1a2fb20055769418a98cbf3d5499c78e76e0daecdb57b080f3148863922c0ab9" Nov 01 23:00:05.020821 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:05.020778 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj_odh-ai-gateway-infra(4f29e1ad-6bae-4ab5-a21c-7c4505e9606a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" Nov 01 23:00:05.022198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.022179 4981 generic.go:356] "Generic (PLEG): container finished" podID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerID="66eb1d03d5665ba0ecc48f6dcc1a7360e2079632e94a6105587a9bbbd9a5454c" exitCode=60 Nov 01 23:00:05.022287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.022244 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" event={"ID":"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec","Type":"ContainerDied","Data":"66eb1d03d5665ba0ecc48f6dcc1a7360e2079632e94a6105587a9bbbd9a5454c"} Nov 01 23:00:05.022531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.022515 4981 scope.go:122] "RemoveContainer" containerID="66eb1d03d5665ba0ecc48f6dcc1a7360e2079632e94a6105587a9bbbd9a5454c" Nov 01 23:00:05.022890 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:05.022867 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n_odh-ai-gateway-infra(85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" Nov 01 23:00:05.024511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.024486 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerStarted","Data":"d0763141a86241ce9339ff54ef538c0cd87db8527295c9ab8a4a96f0c7764097"} Nov 01 23:00:05.034982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.034962 4981 scope.go:122] "RemoveContainer" containerID="43c80411bdb0b9f600169dd301b8441949415ce83e4a0170bc4a7a639cc5641d" Nov 01 23:00:05.053368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.053346 4981 scope.go:122] "RemoveContainer" containerID="50f931b92e815793e1e7f8d5988643c2f44057f329821e6c7d6d762a8f60b93b" Nov 01 23:00:05.070928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.070901 4981 scope.go:122] "RemoveContainer" containerID="8556fb8d2bd314fa4e68ecd4ffd5e058d4edd117e3b5c8e49a00c95d788e3000" Nov 01 23:00:05.134803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.134766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:05.960339 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:05.960302 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:06.077373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.077344 4981 scope.go:122] "RemoveContainer" containerID="3b17429a23c01391aa0e11c3c10cb0cc8206cd27587eb9ac29ce71104ab187a8" Nov 01 23:00:06.077962 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.077796 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b_odh-ai-gateway-infra(28bdfd58-37b7-479b-9633-7ed9e5b288c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" Nov 01 23:00:06.081938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.081915 4981 scope.go:122] "RemoveContainer" containerID="1a2fb20055769418a98cbf3d5499c78e76e0daecdb57b080f3148863922c0ab9" Nov 01 23:00:06.082376 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.082347 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj_odh-ai-gateway-infra(4f29e1ad-6bae-4ab5-a21c-7c4505e9606a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" Nov 01 23:00:06.086454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.086419 4981 scope.go:122] "RemoveContainer" containerID="66eb1d03d5665ba0ecc48f6dcc1a7360e2079632e94a6105587a9bbbd9a5454c" Nov 01 23:00:06.086921 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.086882 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n_odh-ai-gateway-infra(85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" Nov 01 23:00:06.088433 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.088409 4981 generic.go:356] "Generic (PLEG): container finished" podID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerID="d0763141a86241ce9339ff54ef538c0cd87db8527295c9ab8a4a96f0c7764097" exitCode=60 Nov 01 23:00:06.088527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.088491 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerDied","Data":"d0763141a86241ce9339ff54ef538c0cd87db8527295c9ab8a4a96f0c7764097"} Nov 01 23:00:06.088707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.088548 4981 scope.go:122] "RemoveContainer" containerID="6f6083f6144d4666c5023faafd73237c26da83980ab2187366f922fe553a83c7" Nov 01 23:00:06.088816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.088799 4981 scope.go:122] "RemoveContainer" containerID="d0763141a86241ce9339ff54ef538c0cd87db8527295c9ab8a4a96f0c7764097" Nov 01 23:00:06.089202 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.089172 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79_odh-ai-gateway-infra(a9bc57e4-5a18-4f5e-9a24-74750cfc9592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" Nov 01 23:00:06.092975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.092951 4981 generic.go:356] "Generic (PLEG): container finished" podID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerID="1a7b9e11eaf745d1b022a628b6c5c9cec20569e0635fff3f108469f1cf7189b8" exitCode=60 Nov 01 23:00:06.093102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.093039 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerDied","Data":"1a7b9e11eaf745d1b022a628b6c5c9cec20569e0635fff3f108469f1cf7189b8"} Nov 01 23:00:06.093454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.093431 4981 scope.go:122] "RemoveContainer" containerID="1a7b9e11eaf745d1b022a628b6c5c9cec20569e0635fff3f108469f1cf7189b8" Nov 01 23:00:06.094002 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.093964 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw_odh-ai-gateway-infra(4ae1171c-3cf2-43db-b6c1-2a1469875ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" Nov 01 23:00:06.095726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.095695 4981 generic.go:356] "Generic (PLEG): container finished" podID="15a48509-ebff-4841-bea9-eead8afd4a82" containerID="343e4073125bd391d60fdc8cfe7bbbec1eabe49161d27112f9ed1e21d505b174" exitCode=60 Nov 01 23:00:06.095837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.095772 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerDied","Data":"343e4073125bd391d60fdc8cfe7bbbec1eabe49161d27112f9ed1e21d505b174"} Nov 01 23:00:06.096078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.096061 4981 scope.go:122] "RemoveContainer" containerID="343e4073125bd391d60fdc8cfe7bbbec1eabe49161d27112f9ed1e21d505b174" Nov 01 23:00:06.096402 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.096374 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw_odh-ai-gateway-infra(15a48509-ebff-4841-bea9-eead8afd4a82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" Nov 01 23:00:06.098817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.098794 4981 scope.go:122] "RemoveContainer" containerID="fad5cdae5685b74561184c77269d965f48089007a864b5429c48fb3af0b74e54" Nov 01 23:00:06.099157 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.099123 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988_odh-ai-gateway-infra(0350611d-dea7-4f33-88a7-04f0fb719138)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" Nov 01 23:00:06.100650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.100630 4981 generic.go:356] "Generic (PLEG): container finished" podID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerID="337724fa93aff26d135c036d3ddc42c122b15c38a3b0089cc65237c7e0c5e29f" exitCode=60 Nov 01 23:00:06.100768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.100712 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerDied","Data":"337724fa93aff26d135c036d3ddc42c122b15c38a3b0089cc65237c7e0c5e29f"} Nov 01 23:00:06.100912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.100897 4981 scope.go:122] "RemoveContainer" containerID="337724fa93aff26d135c036d3ddc42c122b15c38a3b0089cc65237c7e0c5e29f" Nov 01 23:00:06.101154 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.101132 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9_odh-ai-gateway-infra(f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" Nov 01 23:00:06.102763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.102748 4981 scope.go:122] "RemoveContainer" containerID="aee23b392f56e848678916c05dbcc9318e2b10c99709c888e641b47fc77d5621" Nov 01 23:00:06.103244 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.103015 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq_odh-ai-gateway-infra(f2dad26e-4750-4dcc-b884-93aabfdcd334)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" Nov 01 23:00:06.108469 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.108455 4981 scope.go:122] "RemoveContainer" containerID="85134349c28c858786628bde6313d56e779664897a7b3172d9563efa8478d93a" Nov 01 23:00:06.108647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.108618 4981 generic.go:356] "Generic (PLEG): container finished" podID="a43a9074-cf62-4550-9509-e51ead257c0a" containerID="14dd8960a7f754c04dc58fae816e96072ec45e3a3a9cba8462c5315b46a2eb69" exitCode=60 Nov 01 23:00:06.108741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.108688 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerDied","Data":"14dd8960a7f754c04dc58fae816e96072ec45e3a3a9cba8462c5315b46a2eb69"} Nov 01 23:00:06.109129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.109107 4981 scope.go:122] "RemoveContainer" containerID="14dd8960a7f754c04dc58fae816e96072ec45e3a3a9cba8462c5315b46a2eb69" Nov 01 23:00:06.110002 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.109958 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g_odh-ai-gateway-infra(a43a9074-cf62-4550-9509-e51ead257c0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" Nov 01 23:00:06.111342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.111320 4981 generic.go:356] "Generic (PLEG): container finished" podID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerID="fd65089478afbdfb5b314581081848cc7c377786d694f8211b74caf72b71b0fd" exitCode=60 Nov 01 23:00:06.111464 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.111423 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerDied","Data":"fd65089478afbdfb5b314581081848cc7c377786d694f8211b74caf72b71b0fd"} Nov 01 23:00:06.111793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.111774 4981 scope.go:122] "RemoveContainer" containerID="fd65089478afbdfb5b314581081848cc7c377786d694f8211b74caf72b71b0fd" Nov 01 23:00:06.112237 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.112211 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj_odh-ai-gateway-infra(11e66cad-b147-4b3d-96eb-6b73e5a5144f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" Nov 01 23:00:06.114198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.114179 4981 generic.go:356] "Generic (PLEG): container finished" podID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerID="92576b20ee575e695975e4d25f2a891977c0f96a62b392bd678fdd01a7ba9c78" exitCode=60 Nov 01 23:00:06.114303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.114201 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerDied","Data":"92576b20ee575e695975e4d25f2a891977c0f96a62b392bd678fdd01a7ba9c78"} Nov 01 23:00:06.114540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.114524 4981 scope.go:122] "RemoveContainer" containerID="92576b20ee575e695975e4d25f2a891977c0f96a62b392bd678fdd01a7ba9c78" Nov 01 23:00:06.114950 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.114924 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f_odh-ai-gateway-infra(1cf9a168-f16d-4282-8933-85e42c4893c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" Nov 01 23:00:06.116460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.116438 4981 generic.go:356] "Generic (PLEG): container finished" podID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerID="e345bd906cd0d089424ed92c02403a6ea6206bbad8f5079bf892e19c919decbd" exitCode=60 Nov 01 23:00:06.116550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.116505 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" event={"ID":"090be1a3-6eab-40e2-a821-f84b54f74ff8","Type":"ContainerDied","Data":"e345bd906cd0d089424ed92c02403a6ea6206bbad8f5079bf892e19c919decbd"} Nov 01 23:00:06.116864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.116845 4981 scope.go:122] "RemoveContainer" containerID="e345bd906cd0d089424ed92c02403a6ea6206bbad8f5079bf892e19c919decbd" Nov 01 23:00:06.117244 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:06.117221 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft_odh-ai-gateway-infra(090be1a3-6eab-40e2-a821-f84b54f74ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" Nov 01 23:00:06.129030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.129008 4981 scope.go:122] "RemoveContainer" containerID="cb89cce99eabfa67b5f659682f4462760681dc79351653ad6e188794fe522199" Nov 01 23:00:06.145512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.145485 4981 scope.go:122] "RemoveContainer" containerID="271b7f692b2dc05cecbdacb94162139e52c88e8a6c96b721d391ceab342634a5" Nov 01 23:00:06.163049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.163024 4981 scope.go:122] "RemoveContainer" containerID="a03d0bab573e073d6155902852445fd8c6392af74f7e04c155021ccbe6f69fa8" Nov 01 23:00:06.181784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.181761 4981 scope.go:122] "RemoveContainer" containerID="75374f19c39da0b782defccf99e654bb5baa5ac2a1cd93a38788dc6630c21f59" Nov 01 23:00:06.197973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.197950 4981 scope.go:122] "RemoveContainer" containerID="cca4451f2e878f9359696547375468114020c09020983399f0e2995d97b6c983" Nov 01 23:00:06.220742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.220700 4981 scope.go:122] "RemoveContainer" containerID="7171ee9d6f720403e4a6344a6d6fcb48d963e0b7f541c66eae7a058cf1e5bd72" Nov 01 23:00:06.863710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:06.863674 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:07.132122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.132034 4981 scope.go:122] "RemoveContainer" containerID="1a7b9e11eaf745d1b022a628b6c5c9cec20569e0635fff3f108469f1cf7189b8" Nov 01 23:00:07.132642 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.132500 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw_odh-ai-gateway-infra(4ae1171c-3cf2-43db-b6c1-2a1469875ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" Nov 01 23:00:07.133805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.133783 4981 scope.go:122] "RemoveContainer" containerID="343e4073125bd391d60fdc8cfe7bbbec1eabe49161d27112f9ed1e21d505b174" Nov 01 23:00:07.134201 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.134175 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw_odh-ai-gateway-infra(15a48509-ebff-4841-bea9-eead8afd4a82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" Nov 01 23:00:07.135566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.135549 4981 scope.go:122] "RemoveContainer" containerID="337724fa93aff26d135c036d3ddc42c122b15c38a3b0089cc65237c7e0c5e29f" Nov 01 23:00:07.135921 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.135897 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9_odh-ai-gateway-infra(f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" Nov 01 23:00:07.137142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.137126 4981 scope.go:122] "RemoveContainer" containerID="14dd8960a7f754c04dc58fae816e96072ec45e3a3a9cba8462c5315b46a2eb69" Nov 01 23:00:07.137462 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.137433 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g_odh-ai-gateway-infra(a43a9074-cf62-4550-9509-e51ead257c0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" Nov 01 23:00:07.138763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.138744 4981 scope.go:122] "RemoveContainer" containerID="fd65089478afbdfb5b314581081848cc7c377786d694f8211b74caf72b71b0fd" Nov 01 23:00:07.139033 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.139011 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj_odh-ai-gateway-infra(11e66cad-b147-4b3d-96eb-6b73e5a5144f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" Nov 01 23:00:07.140391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.140372 4981 scope.go:122] "RemoveContainer" containerID="92576b20ee575e695975e4d25f2a891977c0f96a62b392bd678fdd01a7ba9c78" Nov 01 23:00:07.140672 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.140648 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f_odh-ai-gateway-infra(1cf9a168-f16d-4282-8933-85e42c4893c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" Nov 01 23:00:07.141945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.141928 4981 scope.go:122] "RemoveContainer" containerID="e345bd906cd0d089424ed92c02403a6ea6206bbad8f5079bf892e19c919decbd" Nov 01 23:00:07.142204 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.142176 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft_odh-ai-gateway-infra(090be1a3-6eab-40e2-a821-f84b54f74ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" Nov 01 23:00:07.143564 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.143544 4981 scope.go:122] "RemoveContainer" containerID="d0763141a86241ce9339ff54ef538c0cd87db8527295c9ab8a4a96f0c7764097" Nov 01 23:00:07.143816 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:07.143798 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79_odh-ai-gateway-infra(a9bc57e4-5a18-4f5e-9a24-74750cfc9592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" Nov 01 23:00:07.587055 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:07.587000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:08.072884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.072852 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:08.352638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.352545 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q"] Nov 01 23:00:08.353518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.353499 4981 cpu_manager.go:414] "containerMap: removing container" podUID="03fff2db-3e5f-42e6-b407-ded0679a0c1c" containerName="revoke-keys" Nov 01 23:00:08.353571 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.353520 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fff2db-3e5f-42e6-b407-ded0679a0c1c" containerName="revoke-keys" Nov 01 23:00:08.353742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.353731 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fff2db-3e5f-42e6-b407-ded0679a0c1c" containerName="revoke-keys" Nov 01 23:00:08.359374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.359353 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.363701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.363673 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:00:08.366764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.366740 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q"] Nov 01 23:00:08.481593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.481548 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43543bda-f525-4abb-a561-148da424f4e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.481777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.481679 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlsk\" (UniqueName: \"kubernetes.io/projected/43543bda-f525-4abb-a561-148da424f4e4-kube-api-access-gdlsk\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.481825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.481778 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43543bda-f525-4abb-a561-148da424f4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.583304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.583264 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43543bda-f525-4abb-a561-148da424f4e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.583473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.583337 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlsk\" (UniqueName: \"kubernetes.io/projected/43543bda-f525-4abb-a561-148da424f4e4-kube-api-access-gdlsk\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.583473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.583396 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43543bda-f525-4abb-a561-148da424f4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.584126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.584102 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43543bda-f525-4abb-a561-148da424f4e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.585698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.585677 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43543bda-f525-4abb-a561-148da424f4e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.604749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.604680 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlsk\" (UniqueName: \"kubernetes.io/projected/43543bda-f525-4abb-a561-148da424f4e4-kube-api-access-gdlsk\") pod \"maas-api-revoke-keys-e2e-inf-a-021714-s5j9q\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.676678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.676644 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:08.930935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.930898 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:08.978211 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:08.978167 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43543bda_f525_4abb_a561_148da424f4e4.slice/crio-138e5c82c47ea393dacec11c9ab25de4f667569183cf6f39f181dfb90cead96c WatchSource:0}: Error finding container 138e5c82c47ea393dacec11c9ab25de4f667569183cf6f39f181dfb90cead96c: Status 404 returned error can't find the container with id 138e5c82c47ea393dacec11c9ab25de4f667569183cf6f39f181dfb90cead96c Nov 01 23:00:08.986237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:08.985047 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q"] Nov 01 23:00:09.236124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:09.236042 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" event={"ID":"43543bda-f525-4abb-a561-148da424f4e4","Type":"ContainerStarted","Data":"138e5c82c47ea393dacec11c9ab25de4f667569183cf6f39f181dfb90cead96c"} Nov 01 23:00:09.577689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:09.577658 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:10.247618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:10.247571 4981 generic.go:356] "Generic (PLEG): container finished" podID="43543bda-f525-4abb-a561-148da424f4e4" containerID="10c5aeb43fa027dc0eb207262156ece9794ba17b9fecc394cb8ac7fa90d5140b" exitCode=0 Nov 01 23:00:10.247787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:10.247668 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" event={"ID":"43543bda-f525-4abb-a561-148da424f4e4","Type":"ContainerDied","Data":"10c5aeb43fa027dc0eb207262156ece9794ba17b9fecc394cb8ac7fa90d5140b"} Nov 01 23:00:10.293034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:10.293003 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:10.956462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:10.956428 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:11.461358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.461334 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:11.543897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.543793 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43543bda-f525-4abb-a561-148da424f4e4-kube-api-access-gdlsk\" (UniqueName: \"kubernetes.io/projected/43543bda-f525-4abb-a561-148da424f4e4-kube-api-access-gdlsk\") pod \"43543bda-f525-4abb-a561-148da424f4e4\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " Nov 01 23:00:11.544089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.543916 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43543bda-f525-4abb-a561-148da424f4e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43543bda-f525-4abb-a561-148da424f4e4-maas-api-xks-ca\") pod \"43543bda-f525-4abb-a561-148da424f4e4\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " Nov 01 23:00:11.544089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.543979 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43543bda-f525-4abb-a561-148da424f4e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43543bda-f525-4abb-a561-148da424f4e4-maas-api-openshift-ca\") pod \"43543bda-f525-4abb-a561-148da424f4e4\" (UID: \"43543bda-f525-4abb-a561-148da424f4e4\") " Nov 01 23:00:11.545290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.545134 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43543bda-f525-4abb-a561-148da424f4e4-maas-api-openshift-ca" pod "43543bda-f525-4abb-a561-148da424f4e4" (UID: "43543bda-f525-4abb-a561-148da424f4e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:11.547159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.547129 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43543bda-f525-4abb-a561-148da424f4e4-kube-api-access-gdlsk" pod "43543bda-f525-4abb-a561-148da424f4e4" (UID: "43543bda-f525-4abb-a561-148da424f4e4"). InnerVolumeSpecName "kube-api-access-gdlsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:11.547276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.547257 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43543bda-f525-4abb-a561-148da424f4e4-maas-api-xks-ca" pod "43543bda-f525-4abb-a561-148da424f4e4" (UID: "43543bda-f525-4abb-a561-148da424f4e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:11.644954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.644919 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:11.646416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.646392 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdlsk\" (UniqueName: \"kubernetes.io/projected/43543bda-f525-4abb-a561-148da424f4e4-kube-api-access-gdlsk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:11.646503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.646419 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43543bda-f525-4abb-a561-148da424f4e4-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:11.646503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:11.646430 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43543bda-f525-4abb-a561-148da424f4e4-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:12.269876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:12.269845 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" Nov 01 23:00:12.269876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:12.269865 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q" event={"ID":"43543bda-f525-4abb-a561-148da424f4e4","Type":"ContainerDied","Data":"138e5c82c47ea393dacec11c9ab25de4f667569183cf6f39f181dfb90cead96c"} Nov 01 23:00:12.270416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:12.269916 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="138e5c82c47ea393dacec11c9ab25de4f667569183cf6f39f181dfb90cead96c" Nov 01 23:00:12.562326 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:12.562232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:13.499190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:13.499157 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:14.116634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:14.116589 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:14.528586 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:14.528533 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:00:14.828965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:14.828887 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:15.179538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:15.179503 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:15.284299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:15.284261 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:15.744599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:15.744175 4981 scope.go:122] "RemoveContainer" containerID="d0763141a86241ce9339ff54ef538c0cd87db8527295c9ab8a4a96f0c7764097" Nov 01 23:00:15.745124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:15.745033 4981 scope.go:122] "RemoveContainer" containerID="fad5cdae5685b74561184c77269d965f48089007a864b5429c48fb3af0b74e54" Nov 01 23:00:15.843523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:15.825057 4981 scope.go:122] "RemoveContainer" containerID="1a2fb20055769418a98cbf3d5499c78e76e0daecdb57b080f3148863922c0ab9" Nov 01 23:00:16.274825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:16.214139 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf"] Nov 01 23:00:16.613287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:16.613250 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2aa42aa-9qlpf"] Nov 01 23:00:16.613980 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:16.613835 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:16.726928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:16.726887 4981 scope.go:122] "RemoveContainer" containerID="3b17429a23c01391aa0e11c3c10cb0cc8206cd27587eb9ac29ce71104ab187a8" Nov 01 23:00:16.727566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:16.727194 4981 scope.go:122] "RemoveContainer" containerID="66eb1d03d5665ba0ecc48f6dcc1a7360e2079632e94a6105587a9bbbd9a5454c" Nov 01 23:00:16.727836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:16.727761 4981 scope.go:122] "RemoveContainer" containerID="337724fa93aff26d135c036d3ddc42c122b15c38a3b0089cc65237c7e0c5e29f" Nov 01 23:00:16.744316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:16.744283 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fd7304-5f6a-4839-a02e-999f92f44e43" path="/var/lib/kubelet/pods/d4fd7304-5f6a-4839-a02e-999f92f44e43/volumes" Nov 01 23:00:17.407567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.407535 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:17.630144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.630108 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" event={"ID":"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a","Type":"ContainerStarted","Data":"00e4107456aded7eb73a6ae036b0d39270c19b71bca4dfb10ab855aa3409a3f6"} Nov 01 23:00:17.632489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.632463 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" event={"ID":"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec","Type":"ContainerStarted","Data":"a7be93d649d84a419f39217b2af343ce12eb13da17824f44214a62da2751b47a"} Nov 01 23:00:17.634634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.634607 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerStarted","Data":"35d9acd6fd92f2522f710f1a200ba26f105fa399100d6d96db3751b173dc6ff9"} Nov 01 23:00:17.636730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.636707 4981 generic.go:356] "Generic (PLEG): container finished" podID="0350611d-dea7-4f33-88a7-04f0fb719138" containerID="7cc1d8e829518e630b927d538d208b8242f17fc5035b456d622cf85b6b25959f" exitCode=60 Nov 01 23:00:17.636855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.636771 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" event={"ID":"0350611d-dea7-4f33-88a7-04f0fb719138","Type":"ContainerDied","Data":"7cc1d8e829518e630b927d538d208b8242f17fc5035b456d622cf85b6b25959f"} Nov 01 23:00:17.636855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.636815 4981 scope.go:122] "RemoveContainer" containerID="fad5cdae5685b74561184c77269d965f48089007a864b5429c48fb3af0b74e54" Nov 01 23:00:17.637479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.637462 4981 scope.go:122] "RemoveContainer" containerID="7cc1d8e829518e630b927d538d208b8242f17fc5035b456d622cf85b6b25959f" Nov 01 23:00:17.637890 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:17.637864 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988_odh-ai-gateway-infra(0350611d-dea7-4f33-88a7-04f0fb719138)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" Nov 01 23:00:17.639253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.639233 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerStarted","Data":"f206913957d2b2ad9bd6d6a8ddd68706c6e5f5ce8f08a0e47765c28116e917e8"} Nov 01 23:00:17.641423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.641392 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" event={"ID":"28bdfd58-37b7-479b-9633-7ed9e5b288c5","Type":"ContainerStarted","Data":"d75cdeb5be3db808f7e908e24e525d113553761b4070ec3d74aa17e70b2311a1"} Nov 01 23:00:17.727302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.727269 4981 scope.go:122] "RemoveContainer" containerID="343e4073125bd391d60fdc8cfe7bbbec1eabe49161d27112f9ed1e21d505b174" Nov 01 23:00:17.727698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.727677 4981 scope.go:122] "RemoveContainer" containerID="14dd8960a7f754c04dc58fae816e96072ec45e3a3a9cba8462c5315b46a2eb69" Nov 01 23:00:17.727846 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:17.727819 4981 scope.go:122] "RemoveContainer" containerID="fd65089478afbdfb5b314581081848cc7c377786d694f8211b74caf72b71b0fd" Nov 01 23:00:18.601695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.601659 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:18.713537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.712850 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x"] Nov 01 23:00:18.719095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.718419 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj"] Nov 01 23:00:18.728649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.726104 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79"] Nov 01 23:00:18.729519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.729495 4981 scope.go:122] "RemoveContainer" containerID="92576b20ee575e695975e4d25f2a891977c0f96a62b392bd678fdd01a7ba9c78" Nov 01 23:00:18.736731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.732072 4981 scope.go:122] "RemoveContainer" containerID="aee23b392f56e848678916c05dbcc9318e2b10c99709c888e641b47fc77d5621" Nov 01 23:00:18.736731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.732652 4981 generic.go:356] "Generic (PLEG): container finished" podID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerID="f206913957d2b2ad9bd6d6a8ddd68706c6e5f5ce8f08a0e47765c28116e917e8" exitCode=60 Nov 01 23:00:18.751690 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.751528 4981 generic.go:356] "Generic (PLEG): container finished" podID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerID="d75cdeb5be3db808f7e908e24e525d113553761b4070ec3d74aa17e70b2311a1" exitCode=60 Nov 01 23:00:18.760870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.759042 4981 generic.go:356] "Generic (PLEG): container finished" podID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerID="00e4107456aded7eb73a6ae036b0d39270c19b71bca4dfb10ab855aa3409a3f6" exitCode=60 Nov 01 23:00:18.792423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.792386 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:18.797422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797392 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988"] Nov 01 23:00:18.797562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797480 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n"] Nov 01 23:00:18.797562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797543 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerDied","Data":"f206913957d2b2ad9bd6d6a8ddd68706c6e5f5ce8f08a0e47765c28116e917e8"} Nov 01 23:00:18.797695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797657 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerStarted","Data":"0cf81a2a5a88c1028539fceeb26903c54d1ce04e3d104bfc053a7f2ad2613571"} Nov 01 23:00:18.797736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797705 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9"] Nov 01 23:00:18.797774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797748 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerStarted","Data":"2341bcd6f14a25df48d37141c85583d61d00ddf95ff64e185e19ffe8cf3996f3"} Nov 01 23:00:18.797809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797776 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" event={"ID":"28bdfd58-37b7-479b-9633-7ed9e5b288c5","Type":"ContainerDied","Data":"d75cdeb5be3db808f7e908e24e525d113553761b4070ec3d74aa17e70b2311a1"} Nov 01 23:00:18.797860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.797817 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" event={"ID":"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a","Type":"ContainerDied","Data":"00e4107456aded7eb73a6ae036b0d39270c19b71bca4dfb10ab855aa3409a3f6"} Nov 01 23:00:18.798526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.798502 4981 scope.go:122] "RemoveContainer" containerID="d75cdeb5be3db808f7e908e24e525d113553761b4070ec3d74aa17e70b2311a1" Nov 01 23:00:18.799402 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:18.799372 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b_odh-ai-gateway-infra(28bdfd58-37b7-479b-9633-7ed9e5b288c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" Nov 01 23:00:18.800049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.800027 4981 scope.go:122] "RemoveContainer" containerID="337724fa93aff26d135c036d3ddc42c122b15c38a3b0089cc65237c7e0c5e29f" Nov 01 23:00:18.898670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.896534 4981 generic.go:356] "Generic (PLEG): container finished" podID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerID="a7be93d649d84a419f39217b2af343ce12eb13da17824f44214a62da2751b47a" exitCode=60 Nov 01 23:00:18.898670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.896708 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" event={"ID":"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec","Type":"ContainerDied","Data":"a7be93d649d84a419f39217b2af343ce12eb13da17824f44214a62da2751b47a"} Nov 01 23:00:18.910236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.908361 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:18.911338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.911173 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b"] Nov 01 23:00:18.913165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.912539 4981 generic.go:356] "Generic (PLEG): container finished" podID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerID="35d9acd6fd92f2522f710f1a200ba26f105fa399100d6d96db3751b173dc6ff9" exitCode=60 Nov 01 23:00:18.913165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.912862 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerDied","Data":"35d9acd6fd92f2522f710f1a200ba26f105fa399100d6d96db3751b173dc6ff9"} Nov 01 23:00:18.934707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.934009 4981 scope.go:122] "RemoveContainer" containerID="3b17429a23c01391aa0e11c3c10cb0cc8206cd27587eb9ac29ce71104ab187a8" Nov 01 23:00:18.950679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:18.949923 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerStarted","Data":"a0480fcdbd39e983829078950b29547ba57138839eb8cf051811541e158e1201"} Nov 01 23:00:19.015642 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.015607 4981 scope.go:122] "RemoveContainer" containerID="1a2fb20055769418a98cbf3d5499c78e76e0daecdb57b080f3148863922c0ab9" Nov 01 23:00:19.061982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.045766 4981 scope.go:122] "RemoveContainer" containerID="66eb1d03d5665ba0ecc48f6dcc1a7360e2079632e94a6105587a9bbbd9a5454c" Nov 01 23:00:19.418935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.409749 4981 scope.go:122] "RemoveContainer" containerID="d0763141a86241ce9339ff54ef538c0cd87db8527295c9ab8a4a96f0c7764097" Nov 01 23:00:19.423819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.423785 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7"] Nov 01 23:00:19.456677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.456600 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85fae1b3-gzrm7"] Nov 01 23:00:19.597524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.597499 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:19.660979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.660945 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-kube-api-access-8kpbp\" (UniqueName: \"kubernetes.io/projected/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-kube-api-access-8kpbp\") pod \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " Nov 01 23:00:19.662345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.661249 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-maas-api-service-ca\") pod \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\" (UID: \"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b\") " Nov 01 23:00:19.663068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.663040 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-maas-api-service-ca" pod "f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" (UID: "f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:19.695201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.695067 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-kube-api-access-8kpbp" pod "f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" (UID: "f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b"). InnerVolumeSpecName "kube-api-access-8kpbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:19.726901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.726873 4981 scope.go:122] "RemoveContainer" containerID="e345bd906cd0d089424ed92c02403a6ea6206bbad8f5079bf892e19c919decbd" Nov 01 23:00:19.727736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.727715 4981 scope.go:122] "RemoveContainer" containerID="1a7b9e11eaf745d1b022a628b6c5c9cec20569e0635fff3f108469f1cf7189b8" Nov 01 23:00:19.764930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.764870 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kpbp\" (UniqueName: \"kubernetes.io/projected/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-kube-api-access-8kpbp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.764930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.764904 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.767089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.767067 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:19.769512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.769489 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:19.771758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.771735 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:19.783489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.783389 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:19.865676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.865523 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0350611d-dea7-4f33-88a7-04f0fb719138-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0350611d-dea7-4f33-88a7-04f0fb719138-maas-api-service-ca\") pod \"0350611d-dea7-4f33-88a7-04f0fb719138\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " Nov 01 23:00:19.865676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.865605 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-maas-api-service-ca\") pod \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " Nov 01 23:00:19.865676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.865634 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-kube-api-access-qhctw\" (UniqueName: \"kubernetes.io/projected/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-kube-api-access-qhctw\") pod \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\" (UID: \"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec\") " Nov 01 23:00:19.865983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.865829 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-kube-api-access-n6nwd\" (UniqueName: \"kubernetes.io/projected/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-kube-api-access-n6nwd\") pod \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " Nov 01 23:00:19.865983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.865877 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-kube-api-access-s6dv6\" (UniqueName: \"kubernetes.io/projected/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-kube-api-access-s6dv6\") pod \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " Nov 01 23:00:19.866116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.866024 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-maas-api-service-ca\") pod \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\" (UID: \"a9bc57e4-5a18-4f5e-9a24-74750cfc9592\") " Nov 01 23:00:19.866176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.866137 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0350611d-dea7-4f33-88a7-04f0fb719138-kube-api-access-gw57v\" (UniqueName: \"kubernetes.io/projected/0350611d-dea7-4f33-88a7-04f0fb719138-kube-api-access-gw57v\") pod \"0350611d-dea7-4f33-88a7-04f0fb719138\" (UID: \"0350611d-dea7-4f33-88a7-04f0fb719138\") " Nov 01 23:00:19.866319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.866291 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-maas-api-service-ca" pod "85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" (UID: "85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:19.866409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.866318 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-maas-api-service-ca\") pod \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\" (UID: \"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a\") " Nov 01 23:00:19.866644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.866624 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0350611d-dea7-4f33-88a7-04f0fb719138-maas-api-service-ca" pod "0350611d-dea7-4f33-88a7-04f0fb719138" (UID: "0350611d-dea7-4f33-88a7-04f0fb719138"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:19.866903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.866884 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-maas-api-service-ca" pod "a9bc57e4-5a18-4f5e-9a24-74750cfc9592" (UID: "a9bc57e4-5a18-4f5e-9a24-74750cfc9592"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:19.867255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.867235 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0350611d-dea7-4f33-88a7-04f0fb719138-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.867344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.867260 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.867344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.867274 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.870141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.870114 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0350611d-dea7-4f33-88a7-04f0fb719138-kube-api-access-gw57v" pod "0350611d-dea7-4f33-88a7-04f0fb719138" (UID: "0350611d-dea7-4f33-88a7-04f0fb719138"). InnerVolumeSpecName "kube-api-access-gw57v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:19.871144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.871114 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-kube-api-access-n6nwd" pod "a9bc57e4-5a18-4f5e-9a24-74750cfc9592" (UID: "a9bc57e4-5a18-4f5e-9a24-74750cfc9592"). InnerVolumeSpecName "kube-api-access-n6nwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:19.871905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.871731 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-maas-api-service-ca" pod "4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" (UID: "4f29e1ad-6bae-4ab5-a21c-7c4505e9606a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:19.882330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.879788 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-kube-api-access-s6dv6" pod "4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" (UID: "4f29e1ad-6bae-4ab5-a21c-7c4505e9606a"). InnerVolumeSpecName "kube-api-access-s6dv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:19.882330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.880499 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:19.941306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.941268 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g"] Nov 01 23:00:19.954210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.954121 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj"] Nov 01 23:00:19.970374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.970339 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6nwd\" (UniqueName: \"kubernetes.io/projected/a9bc57e4-5a18-4f5e-9a24-74750cfc9592-kube-api-access-n6nwd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.970630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.970614 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6dv6\" (UniqueName: \"kubernetes.io/projected/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-kube-api-access-s6dv6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.970732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.970720 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw57v\" (UniqueName: \"kubernetes.io/projected/0350611d-dea7-4f33-88a7-04f0fb719138-kube-api-access-gw57v\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.970820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.970808 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:19.973388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.973344 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" event={"ID":"0350611d-dea7-4f33-88a7-04f0fb719138","Type":"ContainerDied","Data":"c4b9b219aada14cc2066b7af4fc5cab4e6c26ad38743f6752a8f601d0517e424"} Nov 01 23:00:19.973532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.973425 4981 scope.go:122] "RemoveContainer" containerID="7cc1d8e829518e630b927d538d208b8242f17fc5035b456d622cf85b6b25959f" Nov 01 23:00:19.973610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.973554 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988" Nov 01 23:00:19.977201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.976040 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-kube-api-access-qhctw" pod "85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" (UID: "85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec"). InnerVolumeSpecName "kube-api-access-qhctw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:19.977201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.977181 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" event={"ID":"f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b","Type":"ContainerDied","Data":"42e5113e0de7d13fc374d754dbb4dd6d6bfac94a34c696a73d9df2f89a7c7b8d"} Nov 01 23:00:19.977433 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.977301 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9" Nov 01 23:00:19.987018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.986307 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" event={"ID":"f2dad26e-4750-4dcc-b884-93aabfdcd334","Type":"ContainerStarted","Data":"49c41a84cdf77be3c9969e24f132dba6ecf28fd3aad8cd8e7effa923b0714b03"} Nov 01 23:00:19.991253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.991222 4981 generic.go:356] "Generic (PLEG): container finished" podID="a43a9074-cf62-4550-9509-e51ead257c0a" containerID="0cf81a2a5a88c1028539fceeb26903c54d1ce04e3d104bfc053a7f2ad2613571" exitCode=60 Nov 01 23:00:19.991490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.991466 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerDied","Data":"0cf81a2a5a88c1028539fceeb26903c54d1ce04e3d104bfc053a7f2ad2613571"} Nov 01 23:00:19.998868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:19.998843 4981 scope.go:122] "RemoveContainer" containerID="f206913957d2b2ad9bd6d6a8ddd68706c6e5f5ce8f08a0e47765c28116e917e8" Nov 01 23:00:20.001914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.001795 4981 generic.go:356] "Generic (PLEG): container finished" podID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerID="2341bcd6f14a25df48d37141c85583d61d00ddf95ff64e185e19ffe8cf3996f3" exitCode=60 Nov 01 23:00:20.002193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.002165 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerDied","Data":"2341bcd6f14a25df48d37141c85583d61d00ddf95ff64e185e19ffe8cf3996f3"} Nov 01 23:00:20.009597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.006200 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerStarted","Data":"5bd22495f7e5456b816f890bef1b4088df458f56d9c96d0c669d4db5e02fe87f"} Nov 01 23:00:20.026825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.018959 4981 scope.go:122] "RemoveContainer" containerID="14dd8960a7f754c04dc58fae816e96072ec45e3a3a9cba8462c5315b46a2eb69" Nov 01 23:00:20.026825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.020653 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" event={"ID":"4f29e1ad-6bae-4ab5-a21c-7c4505e9606a","Type":"ContainerDied","Data":"3eb8844e01d3ee16e070e3e9ba7d7bef8ed882651caa8a25ded412fa44e0eb0d"} Nov 01 23:00:20.026825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.020856 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj" Nov 01 23:00:20.026825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.025311 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988"] Nov 01 23:00:20.033380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.033349 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-64b196-29892900-q5988"] Nov 01 23:00:20.072407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.064835 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" event={"ID":"85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec","Type":"ContainerDied","Data":"6b42424c6dea00d0fa0958695d8284aff129eecec72473229d67543e9a41b288"} Nov 01 23:00:20.072407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.064964 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n" Nov 01 23:00:20.086030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.084777 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" event={"ID":"a9bc57e4-5a18-4f5e-9a24-74750cfc9592","Type":"ContainerDied","Data":"3f70cb01fef05806489ddb298801a5441b9c98b53728ef6dd51a277af6c2ceed"} Nov 01 23:00:20.086030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.084988 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79" Nov 01 23:00:20.088088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.088055 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw"] Nov 01 23:00:20.094231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.094208 4981 scope.go:122] "RemoveContainer" containerID="fd65089478afbdfb5b314581081848cc7c377786d694f8211b74caf72b71b0fd" Nov 01 23:00:20.095076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.095051 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhctw\" (UniqueName: \"kubernetes.io/projected/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec-kube-api-access-qhctw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.104625 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.102266 4981 generic.go:356] "Generic (PLEG): container finished" podID="15a48509-ebff-4841-bea9-eead8afd4a82" containerID="a0480fcdbd39e983829078950b29547ba57138839eb8cf051811541e158e1201" exitCode=60 Nov 01 23:00:20.104625 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.102324 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerDied","Data":"a0480fcdbd39e983829078950b29547ba57138839eb8cf051811541e158e1201"} Nov 01 23:00:20.154101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.153710 4981 scope.go:122] "RemoveContainer" containerID="00e4107456aded7eb73a6ae036b0d39270c19b71bca4dfb10ab855aa3409a3f6" Nov 01 23:00:20.229776 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.229724 4981 scope.go:122] "RemoveContainer" containerID="a7be93d649d84a419f39217b2af343ce12eb13da17824f44214a62da2751b47a" Nov 01 23:00:20.237559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.237538 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9"] Nov 01 23:00:20.257618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.254920 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f338e4-29892900-q7bp9"] Nov 01 23:00:20.297882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.297854 4981 scope.go:122] "RemoveContainer" containerID="35d9acd6fd92f2522f710f1a200ba26f105fa399100d6d96db3751b173dc6ff9" Nov 01 23:00:20.317035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.317008 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:20.336811 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.336778 4981 scope.go:122] "RemoveContainer" containerID="343e4073125bd391d60fdc8cfe7bbbec1eabe49161d27112f9ed1e21d505b174" Nov 01 23:00:20.406043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.405976 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79"] Nov 01 23:00:20.424139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.417823 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43a9074-cf62-4550-9509-e51ead257c0a-kube-api-access-l8jbd\" (UniqueName: \"kubernetes.io/projected/a43a9074-cf62-4550-9509-e51ead257c0a-kube-api-access-l8jbd\") pod \"a43a9074-cf62-4550-9509-e51ead257c0a\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " Nov 01 23:00:20.424139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.421220 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43a9074-cf62-4550-9509-e51ead257c0a-kube-api-access-l8jbd" pod "a43a9074-cf62-4550-9509-e51ead257c0a" (UID: "a43a9074-cf62-4550-9509-e51ead257c0a"). InnerVolumeSpecName "kube-api-access-l8jbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.424139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.423690 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43a9074-cf62-4550-9509-e51ead257c0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43a9074-cf62-4550-9509-e51ead257c0a-maas-api-service-ca\") pod \"a43a9074-cf62-4550-9509-e51ead257c0a\" (UID: \"a43a9074-cf62-4550-9509-e51ead257c0a\") " Nov 01 23:00:20.424139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.424094 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43a9074-cf62-4550-9509-e51ead257c0a-maas-api-service-ca" pod "a43a9074-cf62-4550-9509-e51ead257c0a" (UID: "a43a9074-cf62-4550-9509-e51ead257c0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.424962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.424941 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d73b25-29892900-nng79"] Nov 01 23:00:20.429243 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.429100 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8jbd\" (UniqueName: \"kubernetes.io/projected/a43a9074-cf62-4550-9509-e51ead257c0a-kube-api-access-l8jbd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.429243 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.429126 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43a9074-cf62-4550-9509-e51ead257c0a-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.446406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.445970 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj"] Nov 01 23:00:20.463801 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.462721 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c49240-29892900-z8dgj"] Nov 01 23:00:20.495620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.494310 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n"] Nov 01 23:00:20.502277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.502242 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f675d-29892900-85h4n"] Nov 01 23:00:20.564773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.564738 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:20.571355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.571282 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:20.687697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.687664 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:20.750501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.750426 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bdfd58-37b7-479b-9633-7ed9e5b288c5-kube-api-access-nqt4t\" (UniqueName: \"kubernetes.io/projected/28bdfd58-37b7-479b-9633-7ed9e5b288c5-kube-api-access-nqt4t\") pod \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " Nov 01 23:00:20.750501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.750487 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" path="/var/lib/kubelet/pods/0350611d-dea7-4f33-88a7-04f0fb719138/volumes" Nov 01 23:00:20.750904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.750886 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e66cad-b147-4b3d-96eb-6b73e5a5144f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e66cad-b147-4b3d-96eb-6b73e5a5144f-maas-api-service-ca\") pod \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " Nov 01 23:00:20.750994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.750974 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bdfd58-37b7-479b-9633-7ed9e5b288c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bdfd58-37b7-479b-9633-7ed9e5b288c5-maas-api-service-ca\") pod \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\" (UID: \"28bdfd58-37b7-479b-9633-7ed9e5b288c5\") " Nov 01 23:00:20.751099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.751081 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e66cad-b147-4b3d-96eb-6b73e5a5144f-kube-api-access-m2tng\" (UniqueName: \"kubernetes.io/projected/11e66cad-b147-4b3d-96eb-6b73e5a5144f-kube-api-access-m2tng\") pod \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\" (UID: \"11e66cad-b147-4b3d-96eb-6b73e5a5144f\") " Nov 01 23:00:20.751217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.751197 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3695f11f-70be-4db2-ba9d-42ae5c5f5734" path="/var/lib/kubelet/pods/3695f11f-70be-4db2-ba9d-42ae5c5f5734/volumes" Nov 01 23:00:20.751321 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.751302 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e66cad-b147-4b3d-96eb-6b73e5a5144f-maas-api-service-ca" pod "11e66cad-b147-4b3d-96eb-6b73e5a5144f" (UID: "11e66cad-b147-4b3d-96eb-6b73e5a5144f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.751709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.751683 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bdfd58-37b7-479b-9633-7ed9e5b288c5-maas-api-service-ca" pod "28bdfd58-37b7-479b-9633-7ed9e5b288c5" (UID: "28bdfd58-37b7-479b-9633-7ed9e5b288c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.752395 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.752368 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" path="/var/lib/kubelet/pods/4f29e1ad-6bae-4ab5-a21c-7c4505e9606a/volumes" Nov 01 23:00:20.752631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.752561 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28bdfd58-37b7-479b-9633-7ed9e5b288c5-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.753439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.753413 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" path="/var/lib/kubelet/pods/85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec/volumes" Nov 01 23:00:20.754821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.754798 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" path="/var/lib/kubelet/pods/a9bc57e4-5a18-4f5e-9a24-74750cfc9592/volumes" Nov 01 23:00:20.755426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.755395 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" path="/var/lib/kubelet/pods/f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b/volumes" Nov 01 23:00:20.760158 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.760122 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bdfd58-37b7-479b-9633-7ed9e5b288c5-kube-api-access-nqt4t" pod "28bdfd58-37b7-479b-9633-7ed9e5b288c5" (UID: "28bdfd58-37b7-479b-9633-7ed9e5b288c5"). InnerVolumeSpecName "kube-api-access-nqt4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.761770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.761738 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e66cad-b147-4b3d-96eb-6b73e5a5144f-kube-api-access-m2tng" pod "11e66cad-b147-4b3d-96eb-6b73e5a5144f" (UID: "11e66cad-b147-4b3d-96eb-6b73e5a5144f"). InnerVolumeSpecName "kube-api-access-m2tng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.853117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.853085 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a48509-ebff-4841-bea9-eead8afd4a82-kube-api-access-p9f5f\" (UniqueName: \"kubernetes.io/projected/15a48509-ebff-4841-bea9-eead8afd4a82-kube-api-access-p9f5f\") pod \"15a48509-ebff-4841-bea9-eead8afd4a82\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " Nov 01 23:00:20.853523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.853474 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a48509-ebff-4841-bea9-eead8afd4a82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a48509-ebff-4841-bea9-eead8afd4a82-maas-api-service-ca\") pod \"15a48509-ebff-4841-bea9-eead8afd4a82\" (UID: \"15a48509-ebff-4841-bea9-eead8afd4a82\") " Nov 01 23:00:20.853963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.853937 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a48509-ebff-4841-bea9-eead8afd4a82-maas-api-service-ca" pod "15a48509-ebff-4841-bea9-eead8afd4a82" (UID: "15a48509-ebff-4841-bea9-eead8afd4a82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:20.854192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.854170 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e66cad-b147-4b3d-96eb-6b73e5a5144f-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.854192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.854191 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2tng\" (UniqueName: \"kubernetes.io/projected/11e66cad-b147-4b3d-96eb-6b73e5a5144f-kube-api-access-m2tng\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.854384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.854203 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a48509-ebff-4841-bea9-eead8afd4a82-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.854384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.854213 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqt4t\" (UniqueName: \"kubernetes.io/projected/28bdfd58-37b7-479b-9633-7ed9e5b288c5-kube-api-access-nqt4t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:20.855630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.855607 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a48509-ebff-4841-bea9-eead8afd4a82-kube-api-access-p9f5f" pod "15a48509-ebff-4841-bea9-eead8afd4a82" (UID: "15a48509-ebff-4841-bea9-eead8afd4a82"). InnerVolumeSpecName "kube-api-access-p9f5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:20.949016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.948976 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:20.956258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:20.956226 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9f5f\" (UniqueName: \"kubernetes.io/projected/15a48509-ebff-4841-bea9-eead8afd4a82-kube-api-access-p9f5f\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:21.128641 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.119754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:21.235955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.235857 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq"] Nov 01 23:00:21.246608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.236790 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" event={"ID":"15a48509-ebff-4841-bea9-eead8afd4a82","Type":"ContainerDied","Data":"2f87fa355a1fc97456bdcbe28fd25bffc4e147ba0f4269c686be7c7da5d55ea2"} Nov 01 23:00:21.246608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.236850 4981 scope.go:122] "RemoveContainer" containerID="a0480fcdbd39e983829078950b29547ba57138839eb8cf051811541e158e1201" Nov 01 23:00:21.246608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.236983 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw" Nov 01 23:00:21.275408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.272741 4981 generic.go:356] "Generic (PLEG): container finished" podID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerID="49c41a84cdf77be3c9969e24f132dba6ecf28fd3aad8cd8e7effa923b0714b03" exitCode=60 Nov 01 23:00:21.275408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.272971 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" event={"ID":"f2dad26e-4750-4dcc-b884-93aabfdcd334","Type":"ContainerDied","Data":"49c41a84cdf77be3c9969e24f132dba6ecf28fd3aad8cd8e7effa923b0714b03"} Nov 01 23:00:21.281089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.281049 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" event={"ID":"a43a9074-cf62-4550-9509-e51ead257c0a","Type":"ContainerDied","Data":"d0103ce19dce7798ef5e432ee93407a46e5e1d6d93d9e8876de6d955f0c39e51"} Nov 01 23:00:21.281238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.281172 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g" Nov 01 23:00:21.286097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.285995 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" event={"ID":"11e66cad-b147-4b3d-96eb-6b73e5a5144f","Type":"ContainerDied","Data":"e239d12af67b63b11b8e9c26633220c39ec76cab7e40bcbd87492a449be341b1"} Nov 01 23:00:21.286097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.286098 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj" Nov 01 23:00:21.289836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.289805 4981 generic.go:356] "Generic (PLEG): container finished" podID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerID="5bd22495f7e5456b816f890bef1b4088df458f56d9c96d0c669d4db5e02fe87f" exitCode=60 Nov 01 23:00:21.289971 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.289870 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerDied","Data":"5bd22495f7e5456b816f890bef1b4088df458f56d9c96d0c669d4db5e02fe87f"} Nov 01 23:00:21.290434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.290412 4981 scope.go:122] "RemoveContainer" containerID="5bd22495f7e5456b816f890bef1b4088df458f56d9c96d0c669d4db5e02fe87f" Nov 01 23:00:21.291026 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:21.290990 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f_odh-ai-gateway-infra(1cf9a168-f16d-4282-8933-85e42c4893c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" Nov 01 23:00:21.295736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.295692 4981 scope.go:122] "RemoveContainer" containerID="aee23b392f56e848678916c05dbcc9318e2b10c99709c888e641b47fc77d5621" Nov 01 23:00:21.298168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.295852 4981 generic.go:356] "Generic (PLEG): container finished" podID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerID="121f5676e9b094c82a2acd25a49ad52e4e60e23b5ce0f3b54e04f46b3ea803d2" exitCode=60 Nov 01 23:00:21.298168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.295955 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" event={"ID":"090be1a3-6eab-40e2-a821-f84b54f74ff8","Type":"ContainerDied","Data":"121f5676e9b094c82a2acd25a49ad52e4e60e23b5ce0f3b54e04f46b3ea803d2"} Nov 01 23:00:21.298168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.296373 4981 scope.go:122] "RemoveContainer" containerID="121f5676e9b094c82a2acd25a49ad52e4e60e23b5ce0f3b54e04f46b3ea803d2" Nov 01 23:00:21.298168 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:21.297921 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft_odh-ai-gateway-infra(090be1a3-6eab-40e2-a821-f84b54f74ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" Nov 01 23:00:21.305535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.305505 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" event={"ID":"28bdfd58-37b7-479b-9633-7ed9e5b288c5","Type":"ContainerDied","Data":"0ba812459bda958839d5e003ad71fb8d9e766b99ce73963334ff0b2b495e3040"} Nov 01 23:00:21.305738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.305636 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b" Nov 01 23:00:21.318204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.318083 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f"] Nov 01 23:00:21.335666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.335605 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerStarted","Data":"bfa92ed28d4390e4f319e03e9c171c3f6e921b43c4b657141e0188264cbf9d66"} Nov 01 23:00:21.336170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.336143 4981 scope.go:122] "RemoveContainer" containerID="bfa92ed28d4390e4f319e03e9c171c3f6e921b43c4b657141e0188264cbf9d66" Nov 01 23:00:21.336705 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:21.336679 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw_odh-ai-gateway-infra(4ae1171c-3cf2-43db-b6c1-2a1469875ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" Nov 01 23:00:21.337090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.337070 4981 scope.go:122] "RemoveContainer" containerID="0cf81a2a5a88c1028539fceeb26903c54d1ce04e3d104bfc053a7f2ad2613571" Nov 01 23:00:21.360682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.359881 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v"] Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361263 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361288 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361312 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361320 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361333 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361341 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361357 4981 cpu_manager.go:414] "containerMap: removing container" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361365 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361391 4981 cpu_manager.go:414] "containerMap: removing container" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361397 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361408 4981 cpu_manager.go:414] "containerMap: removing container" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361415 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361427 4981 cpu_manager.go:414] "containerMap: removing container" podUID="43543bda-f525-4abb-a561-148da424f4e4" containerName="revoke-keys" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361434 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43543bda-f525-4abb-a561-148da424f4e4" containerName="revoke-keys" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361446 4981 cpu_manager.go:414] "containerMap: removing container" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361452 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361464 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361471 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361495 4981 cpu_manager.go:414] "containerMap: removing container" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361502 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361512 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361531 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361544 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361550 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361563 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361569 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361624 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.361628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361632 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361653 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361660 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361683 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361701 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361722 4981 cpu_manager.go:414] "containerMap: removing container" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361729 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361741 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361747 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361765 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361771 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361780 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361786 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361799 4981 cpu_manager.go:414] "containerMap: removing container" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361807 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361827 4981 cpu_manager.go:414] "containerMap: removing container" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.361835 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362086 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362102 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362110 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362121 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362129 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362139 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362149 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362159 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362173 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362183 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362194 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362204 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362217 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="43543bda-f525-4abb-a561-148da424f4e4" containerName="revoke-keys" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362228 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362238 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362245 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362254 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f29e1ad-6bae-4ab5-a21c-7c4505e9606a" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362265 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362274 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:21.363831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.362284 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:21.369039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.369017 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.378562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.376759 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:00:21.385439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.383400 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v"] Nov 01 23:00:21.405062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.396673 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.405062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.396768 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.405062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.397098 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxb6h\" (UniqueName: \"kubernetes.io/projected/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-kube-api-access-mxb6h\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.405062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.400803 4981 scope.go:122] "RemoveContainer" containerID="2341bcd6f14a25df48d37141c85583d61d00ddf95ff64e185e19ffe8cf3996f3" Nov 01 23:00:21.405062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.404264 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g"] Nov 01 23:00:21.412298 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.412231 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-075771-29892900-xjt2g"] Nov 01 23:00:21.420887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.420825 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw"] Nov 01 23:00:21.423050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.422936 4981 scope.go:122] "RemoveContainer" containerID="92576b20ee575e695975e4d25f2a891977c0f96a62b392bd678fdd01a7ba9c78" Nov 01 23:00:21.428876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.428821 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e26733-29892900-95hvw"] Nov 01 23:00:21.449167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.449111 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b"] Nov 01 23:00:21.459639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.458997 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-021714-29892900-fl86b"] Nov 01 23:00:21.472723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.472692 4981 scope.go:122] "RemoveContainer" containerID="e345bd906cd0d089424ed92c02403a6ea6206bbad8f5079bf892e19c919decbd" Nov 01 23:00:21.495922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.495890 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj"] Nov 01 23:00:21.497853 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.497449 4981 scope.go:122] "RemoveContainer" containerID="d75cdeb5be3db808f7e908e24e525d113553761b4070ec3d74aa17e70b2311a1" Nov 01 23:00:21.500333 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.500308 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxb6h\" (UniqueName: \"kubernetes.io/projected/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-kube-api-access-mxb6h\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.501086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.501062 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.502043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.502024 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.502902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.501929 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.505737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.505715 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6ac182-29892900-bbmpj"] Nov 01 23:00:21.521344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.521313 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.523159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.523128 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxb6h\" (UniqueName: \"kubernetes.io/projected/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-kube-api-access-mxb6h\") pod \"maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.572238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.572207 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:21.604786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.604648 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dad26e-4750-4dcc-b884-93aabfdcd334-kube-api-access-k796q\" (UniqueName: \"kubernetes.io/projected/f2dad26e-4750-4dcc-b884-93aabfdcd334-kube-api-access-k796q\") pod \"f2dad26e-4750-4dcc-b884-93aabfdcd334\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " Nov 01 23:00:21.604957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.604836 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dad26e-4750-4dcc-b884-93aabfdcd334-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dad26e-4750-4dcc-b884-93aabfdcd334-maas-api-service-ca\") pod \"f2dad26e-4750-4dcc-b884-93aabfdcd334\" (UID: \"f2dad26e-4750-4dcc-b884-93aabfdcd334\") " Nov 01 23:00:21.605953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.605483 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dad26e-4750-4dcc-b884-93aabfdcd334-maas-api-service-ca" pod "f2dad26e-4750-4dcc-b884-93aabfdcd334" (UID: "f2dad26e-4750-4dcc-b884-93aabfdcd334"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:21.605953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.605920 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2dad26e-4750-4dcc-b884-93aabfdcd334-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:21.607230 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.607203 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dad26e-4750-4dcc-b884-93aabfdcd334-kube-api-access-k796q" pod "f2dad26e-4750-4dcc-b884-93aabfdcd334" (UID: "f2dad26e-4750-4dcc-b884-93aabfdcd334"). InnerVolumeSpecName "kube-api-access-k796q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:21.624711 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.624680 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:21.708241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.708185 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k796q\" (UniqueName: \"kubernetes.io/projected/f2dad26e-4750-4dcc-b884-93aabfdcd334-kube-api-access-k796q\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:21.726318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.726284 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:21.830562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:21.830522 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:22.022949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.022916 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v"] Nov 01 23:00:22.027028 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:22.026999 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ded17e7_24f9_46e9_94a4_ec1f75aa2aee.slice/crio-325f2d9b555f32d2b4a21ca35c75a56c51b70e752e1841108cb77e0dc214cb14 WatchSource:0}: Error finding container 325f2d9b555f32d2b4a21ca35c75a56c51b70e752e1841108cb77e0dc214cb14: Status 404 returned error can't find the container with id 325f2d9b555f32d2b4a21ca35c75a56c51b70e752e1841108cb77e0dc214cb14 Nov 01 23:00:22.350663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.349157 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" Nov 01 23:00:22.350663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.349164 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq" event={"ID":"f2dad26e-4750-4dcc-b884-93aabfdcd334","Type":"ContainerDied","Data":"e583be60db9cb288a8eeca49edf48ad254037e0075072e84b6c6fe6058053bd7"} Nov 01 23:00:22.350663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.349231 4981 scope.go:122] "RemoveContainer" containerID="49c41a84cdf77be3c9969e24f132dba6ecf28fd3aad8cd8e7effa923b0714b03" Nov 01 23:00:22.353943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.353915 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" event={"ID":"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee","Type":"ContainerStarted","Data":"325f2d9b555f32d2b4a21ca35c75a56c51b70e752e1841108cb77e0dc214cb14"} Nov 01 23:00:22.365557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.365528 4981 generic.go:356] "Generic (PLEG): container finished" podID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerID="bfa92ed28d4390e4f319e03e9c171c3f6e921b43c4b657141e0188264cbf9d66" exitCode=60 Nov 01 23:00:22.366019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.365609 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerDied","Data":"bfa92ed28d4390e4f319e03e9c171c3f6e921b43c4b657141e0188264cbf9d66"} Nov 01 23:00:22.366019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.365661 4981 scope.go:122] "RemoveContainer" containerID="1a7b9e11eaf745d1b022a628b6c5c9cec20569e0635fff3f108469f1cf7189b8" Nov 01 23:00:22.366305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.366283 4981 scope.go:122] "RemoveContainer" containerID="bfa92ed28d4390e4f319e03e9c171c3f6e921b43c4b657141e0188264cbf9d66" Nov 01 23:00:22.366946 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:22.366920 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw_odh-ai-gateway-infra(4ae1171c-3cf2-43db-b6c1-2a1469875ee9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" Nov 01 23:00:22.370276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.370253 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft"] Nov 01 23:00:22.432887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.432840 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq"] Nov 01 23:00:22.446635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.445720 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-39e952-29892900-72nlq"] Nov 01 23:00:22.456481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.455264 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw"] Nov 01 23:00:22.610393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.601251 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:22.632492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.632459 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:22.686723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.686702 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:22.738019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.737986 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf9a168-f16d-4282-8933-85e42c4893c8-kube-api-access-xt775\" (UniqueName: \"kubernetes.io/projected/1cf9a168-f16d-4282-8933-85e42c4893c8-kube-api-access-xt775\") pod \"1cf9a168-f16d-4282-8933-85e42c4893c8\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " Nov 01 23:00:22.738372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.738146 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf9a168-f16d-4282-8933-85e42c4893c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf9a168-f16d-4282-8933-85e42c4893c8-maas-api-service-ca\") pod \"1cf9a168-f16d-4282-8933-85e42c4893c8\" (UID: \"1cf9a168-f16d-4282-8933-85e42c4893c8\") " Nov 01 23:00:22.738372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.738284 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090be1a3-6eab-40e2-a821-f84b54f74ff8-kube-api-access-jvmmx\" (UniqueName: \"kubernetes.io/projected/090be1a3-6eab-40e2-a821-f84b54f74ff8-kube-api-access-jvmmx\") pod \"090be1a3-6eab-40e2-a821-f84b54f74ff8\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " Nov 01 23:00:22.738462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.738389 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090be1a3-6eab-40e2-a821-f84b54f74ff8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/090be1a3-6eab-40e2-a821-f84b54f74ff8-maas-api-service-ca\") pod \"090be1a3-6eab-40e2-a821-f84b54f74ff8\" (UID: \"090be1a3-6eab-40e2-a821-f84b54f74ff8\") " Nov 01 23:00:22.739162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.739127 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf9a168-f16d-4282-8933-85e42c4893c8-maas-api-service-ca" pod "1cf9a168-f16d-4282-8933-85e42c4893c8" (UID: "1cf9a168-f16d-4282-8933-85e42c4893c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:22.739310 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.739220 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090be1a3-6eab-40e2-a821-f84b54f74ff8-maas-api-service-ca" pod "090be1a3-6eab-40e2-a821-f84b54f74ff8" (UID: "090be1a3-6eab-40e2-a821-f84b54f74ff8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:22.741168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.741140 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090be1a3-6eab-40e2-a821-f84b54f74ff8-kube-api-access-jvmmx" pod "090be1a3-6eab-40e2-a821-f84b54f74ff8" (UID: "090be1a3-6eab-40e2-a821-f84b54f74ff8"). InnerVolumeSpecName "kube-api-access-jvmmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:22.741849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.741817 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf9a168-f16d-4282-8933-85e42c4893c8-kube-api-access-xt775" pod "1cf9a168-f16d-4282-8933-85e42c4893c8" (UID: "1cf9a168-f16d-4282-8933-85e42c4893c8"). InnerVolumeSpecName "kube-api-access-xt775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:22.744200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.744175 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" path="/var/lib/kubelet/pods/11e66cad-b147-4b3d-96eb-6b73e5a5144f/volumes" Nov 01 23:00:22.744715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.744698 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" path="/var/lib/kubelet/pods/15a48509-ebff-4841-bea9-eead8afd4a82/volumes" Nov 01 23:00:22.745172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.745156 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" path="/var/lib/kubelet/pods/28bdfd58-37b7-479b-9633-7ed9e5b288c5/volumes" Nov 01 23:00:22.745923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.745904 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" path="/var/lib/kubelet/pods/a43a9074-cf62-4550-9509-e51ead257c0a/volumes" Nov 01 23:00:22.746348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.746318 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" path="/var/lib/kubelet/pods/f2dad26e-4750-4dcc-b884-93aabfdcd334/volumes" Nov 01 23:00:22.840762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.840738 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/090be1a3-6eab-40e2-a821-f84b54f74ff8-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:22.840762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.840762 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt775\" (UniqueName: \"kubernetes.io/projected/1cf9a168-f16d-4282-8933-85e42c4893c8-kube-api-access-xt775\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:22.840965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.840774 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cf9a168-f16d-4282-8933-85e42c4893c8-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:22.840965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:22.840784 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmmx\" (UniqueName: \"kubernetes.io/projected/090be1a3-6eab-40e2-a821-f84b54f74ff8-kube-api-access-jvmmx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.356859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.356826 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:23.377242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.377210 4981 generic.go:356] "Generic (PLEG): container finished" podID="8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" containerID="233ab7b80b03339f629a8cfcf9d336f257e90d8dfa06ff73173661ef874460c6" exitCode=0 Nov 01 23:00:23.377442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.377293 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" event={"ID":"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee","Type":"ContainerDied","Data":"233ab7b80b03339f629a8cfcf9d336f257e90d8dfa06ff73173661ef874460c6"} Nov 01 23:00:23.378909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.378889 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" Nov 01 23:00:23.379036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.378930 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f" event={"ID":"1cf9a168-f16d-4282-8933-85e42c4893c8","Type":"ContainerDied","Data":"8f4ad9bad5316cc6b18c9ab7bc2d1feea7d91761f0a8482d2051365d4ce99c46"} Nov 01 23:00:23.379036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.378988 4981 scope.go:122] "RemoveContainer" containerID="5bd22495f7e5456b816f890bef1b4088df458f56d9c96d0c669d4db5e02fe87f" Nov 01 23:00:23.380559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.380540 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" Nov 01 23:00:23.380709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.380609 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft" event={"ID":"090be1a3-6eab-40e2-a821-f84b54f74ff8","Type":"ContainerDied","Data":"ee4bd05cb9d77f145770bc5693cb337b6d1c4c0963c6368807e8a1ff79296bea"} Nov 01 23:00:23.396954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.396927 4981 scope.go:122] "RemoveContainer" containerID="121f5676e9b094c82a2acd25a49ad52e4e60e23b5ce0f3b54e04f46b3ea803d2" Nov 01 23:00:23.491490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.488972 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft"] Nov 01 23:00:23.496627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.494761 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9ad303-29892900-4p9ft"] Nov 01 23:00:23.525679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.525649 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f"] Nov 01 23:00:23.534515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.534485 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-e5f47e-29892900-vmw8f"] Nov 01 23:00:23.663225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.663198 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:23.713341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.713304 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerName="payload-pre-processing" containerID="cri-o://dd379040ccef9eb0d06f9cfa63016765be4badb5b8dcb60f884866172b9d26ff" gracePeriod=25 Nov 01 23:00:23.767305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.767230 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-kube-api-access-j76m7\" (UniqueName: \"kubernetes.io/projected/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-kube-api-access-j76m7\") pod \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " Nov 01 23:00:23.767788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.767322 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-maas-api-service-ca\") pod \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\" (UID: \"4ae1171c-3cf2-43db-b6c1-2a1469875ee9\") " Nov 01 23:00:23.767788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.767736 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-maas-api-service-ca" pod "4ae1171c-3cf2-43db-b6c1-2a1469875ee9" (UID: "4ae1171c-3cf2-43db-b6c1-2a1469875ee9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:23.768132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.768111 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:23.769343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.769319 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-kube-api-access-j76m7" pod "4ae1171c-3cf2-43db-b6c1-2a1469875ee9" (UID: "4ae1171c-3cf2-43db-b6c1-2a1469875ee9"). InnerVolumeSpecName "kube-api-access-j76m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:23.869410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:23.869368 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76m7\" (UniqueName: \"kubernetes.io/projected/4ae1171c-3cf2-43db-b6c1-2a1469875ee9-kube-api-access-j76m7\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:24.160958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.160827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:24.395883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.395857 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" Nov 01 23:00:24.396053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.395908 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw" event={"ID":"4ae1171c-3cf2-43db-b6c1-2a1469875ee9","Type":"ContainerDied","Data":"c11ae6a94be65005954ee3d0e7ef15d204d1c60162c7fb42e74af6374cd3f404"} Nov 01 23:00:24.396053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.395977 4981 scope.go:122] "RemoveContainer" containerID="bfa92ed28d4390e4f319e03e9c171c3f6e921b43c4b657141e0188264cbf9d66" Nov 01 23:00:24.424745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.424719 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw"] Nov 01 23:00:24.431777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.431744 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-53dd05-29892900-bdpkw"] Nov 01 23:00:24.591872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.591843 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:24.683282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.683205 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-kube-api-access-mxb6h\" (UniqueName: \"kubernetes.io/projected/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-kube-api-access-mxb6h\") pod \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " Nov 01 23:00:24.683282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.683276 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-openshift-ca\") pod \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " Nov 01 23:00:24.683470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.683319 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-xks-ca\") pod \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\" (UID: \"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee\") " Nov 01 23:00:24.683731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.683702 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-openshift-ca" pod "8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" (UID: "8ded17e7-24f9-46e9-94a4-ec1f75aa2aee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:24.684175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.684156 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:24.685330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.685307 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-xks-ca" pod "8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" (UID: "8ded17e7-24f9-46e9-94a4-ec1f75aa2aee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:24.685421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.685350 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-kube-api-access-mxb6h" pod "8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" (UID: "8ded17e7-24f9-46e9-94a4-ec1f75aa2aee"). InnerVolumeSpecName "kube-api-access-mxb6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:24.740624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.740591 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" path="/var/lib/kubelet/pods/090be1a3-6eab-40e2-a821-f84b54f74ff8/volumes" Nov 01 23:00:24.741143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.741128 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" path="/var/lib/kubelet/pods/1cf9a168-f16d-4282-8933-85e42c4893c8/volumes" Nov 01 23:00:24.741567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.741553 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" path="/var/lib/kubelet/pods/4ae1171c-3cf2-43db-b6c1-2a1469875ee9/volumes" Nov 01 23:00:24.785864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.785831 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxb6h\" (UniqueName: \"kubernetes.io/projected/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-kube-api-access-mxb6h\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:24.785864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.785864 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:24.959509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:24.959420 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:25.409623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:25.409529 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" Nov 01 23:00:25.409623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:25.409562 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v" event={"ID":"8ded17e7-24f9-46e9-94a4-ec1f75aa2aee","Type":"ContainerDied","Data":"325f2d9b555f32d2b4a21ca35c75a56c51b70e752e1841108cb77e0dc214cb14"} Nov 01 23:00:25.409836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:25.409659 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="325f2d9b555f32d2b4a21ca35c75a56c51b70e752e1841108cb77e0dc214cb14" Nov 01 23:00:25.972758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:25.972728 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:26.727991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:26.727958 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:27.031958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:27.031883 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s"] Nov 01 23:00:27.038255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:27.038228 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ea4404d-xrd8s"] Nov 01 23:00:27.496531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:27.496497 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:27.713754 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:27.713716 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:28.243605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:28.243554 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:28.739013 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:28.738975 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150f1e7b-47c7-4f19-896b-7247aadcb0e0" path="/var/lib/kubelet/pods/150f1e7b-47c7-4f19-896b-7247aadcb0e0/volumes" Nov 01 23:00:29.039393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:29.039300 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx"] Nov 01 23:00:29.053025 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:29.052991 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59afc768-nkjkx"] Nov 01 23:00:29.070592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:29.070556 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:29.864548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:29.864510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:30.738200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:30.738166 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb0284b-633b-47f0-a13a-fb8aa9f38fce" path="/var/lib/kubelet/pods/afb0284b-633b-47f0-a13a-fb8aa9f38fce/volumes" Nov 01 23:00:30.774049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:30.774016 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:31.480632 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:31.480595 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb"] Nov 01 23:00:31.496179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:31.496140 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:31.499287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:31.499263 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft"] Nov 01 23:00:31.598111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:31.598079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:32.369402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:32.369372 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:33.075927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:33.075889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:33.628409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:33.628373 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4"] Nov 01 23:00:33.628737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:33.628703 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" podUID="6fd491ae-556d-4ce8-a5ae-05226714e9ef" containerName="maas-api" containerID="cri-o://2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813" gracePeriod=30 Nov 01 23:00:33.659986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:33.659916 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:33.792927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:33.792889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:33.986615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:33.954705 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:34.074764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.074200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:34.074764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.074312 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q"] Nov 01 23:00:34.136599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.135789 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 23:00:34.141351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.139757 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-021714-s5j9q"] Nov 01 23:00:34.309857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.309766 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd491ae-556d-4ce8-a5ae-05226714e9ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd491ae-556d-4ce8-a5ae-05226714e9ef-openshift-service-ca\") pod \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " Nov 01 23:00:34.351156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.351119 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-metrics-certs\") pod \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " Nov 01 23:00:34.351398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.351211 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd491ae-556d-4ce8-a5ae-05226714e9ef-kube-api-access-t758n\" (UniqueName: \"kubernetes.io/projected/6fd491ae-556d-4ce8-a5ae-05226714e9ef-kube-api-access-t758n\") pod \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " Nov 01 23:00:34.351398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.351273 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls\") pod \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\" (UID: \"6fd491ae-556d-4ce8-a5ae-05226714e9ef\") " Nov 01 23:00:34.354654 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.313300 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd491ae-556d-4ce8-a5ae-05226714e9ef-openshift-service-ca" pod "6fd491ae-556d-4ce8-a5ae-05226714e9ef" (UID: "6fd491ae-556d-4ce8-a5ae-05226714e9ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:34.357789 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.357761 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd491ae-556d-4ce8-a5ae-05226714e9ef-kube-api-access-t758n" pod "6fd491ae-556d-4ce8-a5ae-05226714e9ef" (UID: "6fd491ae-556d-4ce8-a5ae-05226714e9ef"). InnerVolumeSpecName "kube-api-access-t758n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:34.359615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.358958 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls" pod "6fd491ae-556d-4ce8-a5ae-05226714e9ef" (UID: "6fd491ae-556d-4ce8-a5ae-05226714e9ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:34.359736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.359673 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-metrics-certs" pod "6fd491ae-556d-4ce8-a5ae-05226714e9ef" (UID: "6fd491ae-556d-4ce8-a5ae-05226714e9ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:34.369109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.369074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:34.453526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.453496 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:34.453526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.453524 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t758n\" (UniqueName: \"kubernetes.io/projected/6fd491ae-556d-4ce8-a5ae-05226714e9ef-kube-api-access-t758n\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:34.453526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.453534 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fd491ae-556d-4ce8-a5ae-05226714e9ef-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:34.453794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.453546 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fd491ae-556d-4ce8-a5ae-05226714e9ef-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:34.503746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.503713 4981 generic.go:356] "Generic (PLEG): container finished" podID="6fd491ae-556d-4ce8-a5ae-05226714e9ef" containerID="2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813" exitCode=0 Nov 01 23:00:34.503928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.503748 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" event={"ID":"6fd491ae-556d-4ce8-a5ae-05226714e9ef","Type":"ContainerDied","Data":"2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813"} Nov 01 23:00:34.503928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.503803 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" event={"ID":"6fd491ae-556d-4ce8-a5ae-05226714e9ef","Type":"ContainerDied","Data":"9c392d0dd6353dcbcce8f021938d48e1c30932f28c3ce6b542871a21ea3e797b"} Nov 01 23:00:34.503928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.503826 4981 scope.go:122] "RemoveContainer" containerID="2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813" Nov 01 23:00:34.503928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.503764 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4" Nov 01 23:00:34.515756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.515733 4981 scope.go:122] "RemoveContainer" containerID="2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813" Nov 01 23:00:34.516029 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:34.516011 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813\": container with ID starting with 2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813 not found: ID does not exist" containerID="2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813" Nov 01 23:00:34.516093 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.516036 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813"} err="failed to get container status \"2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813\": rpc error: code = NotFound desc = could not find container \"2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813\": container with ID starting with 2d5d6fdafb72e2e3c292ace2986f7c2a6bc451b707e8ff65867aec5901020813 not found: ID does not exist" Nov 01 23:00:34.550339 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.550305 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4"] Nov 01 23:00:34.555524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.555502 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-021714-85885b87f5-xvxf4"] Nov 01 23:00:34.738808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.738777 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43543bda-f525-4abb-a561-148da424f4e4" path="/var/lib/kubelet/pods/43543bda-f525-4abb-a561-148da424f4e4/volumes" Nov 01 23:00:34.739260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:34.739244 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd491ae-556d-4ce8-a5ae-05226714e9ef" path="/var/lib/kubelet/pods/6fd491ae-556d-4ce8-a5ae-05226714e9ef/volumes" Nov 01 23:00:35.070924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:35.070837 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:35.871985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:35.871952 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:36.481858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:36.481812 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" podUID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" containerName="payload-processing" containerID="cri-o://eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546" gracePeriod=25 Nov 01 23:00:36.499896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:36.499838 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" podUID="369ead52-8316-4afd-9b8f-ab992cd76b7d" containerName="payload-pre-processing" containerID="cri-o://aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65" gracePeriod=25 Nov 01 23:00:36.568187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:36.568154 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:37.280609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:37.280547 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:37.712040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:37.711999 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:38.019284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:38.019200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:38.664512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:38.664477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:39.333088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.333053 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd"] Nov 01 23:00:39.333989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.333972 4981 cpu_manager.go:414] "containerMap: removing container" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.333992 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334006 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334012 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334021 4981 cpu_manager.go:414] "containerMap: removing container" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334027 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334040 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334045 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334064 4981 cpu_manager.go:414] "containerMap: removing container" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334069 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334078 4981 cpu_manager.go:414] "containerMap: removing container" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:39.334078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334083 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334093 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334098 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334106 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334111 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334122 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334129 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334144 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6fd491ae-556d-4ce8-a5ae-05226714e9ef" containerName="maas-api" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334149 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd491ae-556d-4ce8-a5ae-05226714e9ef" containerName="maas-api" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334158 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334162 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334175 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334179 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334186 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334191 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334205 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334209 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334220 4981 cpu_manager.go:414] "containerMap: removing container" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334224 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334233 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" containerName="revoke-keys" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334238 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" containerName="revoke-keys" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334248 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334252 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334261 4981 cpu_manager.go:414] "containerMap: removing container" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334265 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334390 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd491ae-556d-4ce8-a5ae-05226714e9ef" containerName="maas-api" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334400 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334410 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334415 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334422 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0350611d-dea7-4f33-88a7-04f0fb719138" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334429 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334436 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" containerName="revoke-keys" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334443 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bc57e4-5a18-4f5e-9a24-74750cfc9592" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334450 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a48509-ebff-4841-bea9-eead8afd4a82" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334456 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334462 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e66cad-b147-4b3d-96eb-6b73e5a5144f" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334468 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78d0d3e-6f2d-4986-8c07-eb9fb66f8b9b" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334474 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334480 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43a9074-cf62-4550-9509-e51ead257c0a" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334488 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334492 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="85bae0ed-a8fd-442f-aaf6-acd7aa5fbfec" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334501 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334507 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bdfd58-37b7-479b-9633-7ed9e5b288c5" containerName="cleanup" Nov 01 23:00:39.334622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.334512 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:39.339996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.339974 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.352511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.352488 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d73b25-openshift-default-dockercfg-xl57x\"" Nov 01 23:00:39.357279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.357252 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd"] Nov 01 23:00:39.473819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.473774 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-podinfo\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.473925 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istiod-ca-cert\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.473968 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-certs\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.474002 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-envoy\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.474151 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-token\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.474215 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvz5n\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-kube-api-access-bvz5n\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474320 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.474300 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-data\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.474353 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.474457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.474440 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-credential-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.485234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.485203 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:39.581766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.581728 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-credential-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.581872 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-podinfo\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582011 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istiod-ca-cert\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582280 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-credential-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582527 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-certs\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582740 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istiod-ca-cert\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582782 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-certs\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582831 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-envoy\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.582944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582906 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-token\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.583005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.582977 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvz5n\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-kube-api-access-bvz5n\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.583068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.583028 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-data\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.583160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.583080 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.583466 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.583438 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-data\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.583592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.583471 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-socket\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.584842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.584823 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-podinfo\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.585260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.585235 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-envoy\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.592130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.592103 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-token\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.592515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.592493 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvz5n\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-kube-api-access-bvz5n\") pod \"e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.658590 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.658554 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:39.955216 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:39.955183 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5ac8ef_364e_4fd4_a832_db34cebe5282.slice/crio-f0861e2f8cbd29026954dc43e466fe0faeaeaa791755487f10185a864aa2c123 WatchSource:0}: Error finding container f0861e2f8cbd29026954dc43e466fe0faeaeaa791755487f10185a864aa2c123: Status 404 returned error can't find the container with id f0861e2f8cbd29026954dc43e466fe0faeaeaa791755487f10185a864aa2c123 Nov 01 23:00:39.956348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.956320 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd"] Nov 01 23:00:39.958448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.958410 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:00:39.958549 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.958495 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:00:39.958549 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:39.958539 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:00:40.320337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:40.319716 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:40.603049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:40.602971 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" event={"ID":"7d5ac8ef-364e-4fd4-a832-db34cebe5282","Type":"ContainerStarted","Data":"715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11"} Nov 01 23:00:40.603193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:40.603061 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" event={"ID":"7d5ac8ef-364e-4fd4-a832-db34cebe5282","Type":"ContainerStarted","Data":"f0861e2f8cbd29026954dc43e466fe0faeaeaa791755487f10185a864aa2c123"} Nov 01 23:00:40.629722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:40.629667 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podStartSLOduration=1.629653886 podStartE2EDuration="1.629653886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:40.624814131 +0000 UTC m=+2206.052692073" watchObservedRunningTime="2026-11-01 23:00:40.629653886 +0000 UTC m=+2206.057531816" Nov 01 23:00:40.658935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:40.658903 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:41.070139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.070103 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:41.170940 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.170880 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:41.459669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.459636 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd"] Nov 01 23:00:41.502274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.502241 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:41.629802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.629762 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4"] Nov 01 23:00:41.797398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.797275 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 01 23:00:41.861472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.851722 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" containerID="cri-o://96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025" gracePeriod=30 Nov 01 23:00:41.946479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.856640 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" containerID="cri-o://b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0" gracePeriod=30 Nov 01 23:00:41.947644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.856684 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 01 23:00:41.947826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.888149 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb"] Nov 01 23:00:41.948008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.947994 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn"] Nov 01 23:00:41.948129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.948110 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n"] Nov 01 23:00:41.948245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.948235 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577"] Nov 01 23:00:41.948346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.948336 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl"] Nov 01 23:00:41.948643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.948618 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" containerID="cri-o://f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1" gracePeriod=30 Nov 01 23:00:41.948934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.948905 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" containerID="cri-o://053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871" gracePeriod=30 Nov 01 23:00:41.949124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.949093 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" containerID="cri-o://3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec" gracePeriod=30 Nov 01 23:00:41.949410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.949378 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" containerID="cri-o://d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957" gracePeriod=30 Nov 01 23:00:41.951418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.949418 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" containerID="cri-o://21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6" gracePeriod=30 Nov 01 23:00:41.972239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.971258 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs"] Nov 01 23:00:41.972533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.972514 4981 cpu_manager.go:414] "containerMap: removing container" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:41.972628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.972534 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090be1a3-6eab-40e2-a821-f84b54f74ff8" containerName="cleanup" Nov 01 23:00:41.972682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.972633 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:41.972682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.972639 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:41.972771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.972761 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae1171c-3cf2-43db-b6c1-2a1469875ee9" containerName="cleanup" Nov 01 23:00:41.972804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.972775 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dad26e-4750-4dcc-b884-93aabfdcd334" containerName="cleanup" Nov 01 23:00:41.972804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:41.972783 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf9a168-f16d-4282-8933-85e42c4893c8" containerName="cleanup" Nov 01 23:00:42.008600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.008112 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.021028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.020228 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs"] Nov 01 23:00:42.099697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.097792 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:00:42.099697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.098540 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:00:42.158155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.158122 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.158155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.158158 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.158523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.158300 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jplkc\" (UniqueName: \"kubernetes.io/projected/fdce05b3-643e-4901-b9dd-027049a2a4c7-kube-api-access-jplkc\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.260424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.260386 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.260616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.260436 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.260616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.260498 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplkc\" (UniqueName: \"kubernetes.io/projected/fdce05b3-643e-4901-b9dd-027049a2a4c7-kube-api-access-jplkc\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.261154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.261130 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.262959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.262936 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.268241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.268221 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jplkc\" (UniqueName: \"kubernetes.io/projected/fdce05b3-643e-4901-b9dd-027049a2a4c7-kube-api-access-jplkc\") pod \"maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.336495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.336458 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:42.650158 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:42.650117 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdce05b3_643e_4901_b9dd_027049a2a4c7.slice/crio-51830f0f54273d16e78b70e7236a289973a50b4e12e2b83dab44ae5e0477866f WatchSource:0}: Error finding container 51830f0f54273d16e78b70e7236a289973a50b4e12e2b83dab44ae5e0477866f: Status 404 returned error can't find the container with id 51830f0f54273d16e78b70e7236a289973a50b4e12e2b83dab44ae5e0477866f Nov 01 23:00:42.651050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.651019 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs"] Nov 01 23:00:42.659408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.659383 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 01 23:00:42.659502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.659432 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 01 23:00:42.685613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.673568 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:42.953947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:42.953908 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" event={"ID":"fdce05b3-643e-4901-b9dd-027049a2a4c7","Type":"ContainerStarted","Data":"51830f0f54273d16e78b70e7236a289973a50b4e12e2b83dab44ae5e0477866f"} Nov 01 23:00:43.359832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.359797 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" podUID="369ead52-8316-4afd-9b8f-ab992cd76b7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:43.659222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.659139 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 01 23:00:43.659222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.659194 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 01 23:00:43.729567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.729536 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:43.970196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.969237 4981 generic.go:356] "Generic (PLEG): container finished" podID="fdce05b3-643e-4901-b9dd-027049a2a4c7" containerID="4da3f1f9163f832a42ed5d522989b33c0a9f690a96fd425070a99a16cb9359e3" exitCode=0 Nov 01 23:00:43.970196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.969780 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" event={"ID":"fdce05b3-643e-4901-b9dd-027049a2a4c7","Type":"ContainerDied","Data":"4da3f1f9163f832a42ed5d522989b33c0a9f690a96fd425070a99a16cb9359e3"} Nov 01 23:00:43.972182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.972155 4981 generic.go:356] "Generic (PLEG): container finished" podID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerID="dd379040ccef9eb0d06f9cfa63016765be4badb5b8dcb60f884866172b9d26ff" exitCode=0 Nov 01 23:00:43.972413 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:43.972396 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" event={"ID":"c3c9e058-877f-41e0-8e59-ac5b17f987fc","Type":"ContainerDied","Data":"dd379040ccef9eb0d06f9cfa63016765be4badb5b8dcb60f884866172b9d26ff"} Nov 01 23:00:44.108173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.108142 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 23:00:44.131228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.131189 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c9e058-877f-41e0-8e59-ac5b17f987fc-kube-api-access-88lpt\" (UniqueName: \"kubernetes.io/projected/c3c9e058-877f-41e0-8e59-ac5b17f987fc-kube-api-access-88lpt\") pod \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " Nov 01 23:00:44.131406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.131250 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c9e058-877f-41e0-8e59-ac5b17f987fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c9e058-877f-41e0-8e59-ac5b17f987fc-plugins-config-volume\") pod \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\" (UID: \"c3c9e058-877f-41e0-8e59-ac5b17f987fc\") " Nov 01 23:00:44.131728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.131702 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c9e058-877f-41e0-8e59-ac5b17f987fc-plugins-config-volume" pod "c3c9e058-877f-41e0-8e59-ac5b17f987fc" (UID: "c3c9e058-877f-41e0-8e59-ac5b17f987fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:44.131885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.131871 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c9e058-877f-41e0-8e59-ac5b17f987fc-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:44.133544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.133521 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c9e058-877f-41e0-8e59-ac5b17f987fc-kube-api-access-88lpt" pod "c3c9e058-877f-41e0-8e59-ac5b17f987fc" (UID: "c3c9e058-877f-41e0-8e59-ac5b17f987fc"). InnerVolumeSpecName "kube-api-access-88lpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:44.233784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.233751 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88lpt\" (UniqueName: \"kubernetes.io/projected/c3c9e058-877f-41e0-8e59-ac5b17f987fc-kube-api-access-88lpt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:44.399811 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.399764 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" podUID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:44.530344 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:44.530263 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:00:44.659875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.659842 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 01 23:00:44.660031 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.659904 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 01 23:00:44.858422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:44.858342 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:45.008884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.008855 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" Nov 01 23:00:45.009089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.008899 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x" event={"ID":"c3c9e058-877f-41e0-8e59-ac5b17f987fc","Type":"ContainerDied","Data":"f179e7de7d5778b6c16ef65da37fc5398c61ee842bf6da91c20235324a8f3149"} Nov 01 23:00:45.009089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.008965 4981 scope.go:122] "RemoveContainer" containerID="dd379040ccef9eb0d06f9cfa63016765be4badb5b8dcb60f884866172b9d26ff" Nov 01 23:00:45.040705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.040673 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x"] Nov 01 23:00:45.047819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.047791 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-021714-7457dbc959-r246x"] Nov 01 23:00:45.272306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.272271 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:45.350920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.350890 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-openshift-ca\") pod \"fdce05b3-643e-4901-b9dd-027049a2a4c7\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " Nov 01 23:00:45.351133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.350944 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-xks-ca\") pod \"fdce05b3-643e-4901-b9dd-027049a2a4c7\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " Nov 01 23:00:45.351278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.351252 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-openshift-ca" pod "fdce05b3-643e-4901-b9dd-027049a2a4c7" (UID: "fdce05b3-643e-4901-b9dd-027049a2a4c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:45.351357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.351264 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdce05b3-643e-4901-b9dd-027049a2a4c7-kube-api-access-jplkc\" (UniqueName: \"kubernetes.io/projected/fdce05b3-643e-4901-b9dd-027049a2a4c7-kube-api-access-jplkc\") pod \"fdce05b3-643e-4901-b9dd-027049a2a4c7\" (UID: \"fdce05b3-643e-4901-b9dd-027049a2a4c7\") " Nov 01 23:00:45.352188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.352160 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:45.353364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.353340 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-xks-ca" pod "fdce05b3-643e-4901-b9dd-027049a2a4c7" (UID: "fdce05b3-643e-4901-b9dd-027049a2a4c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:45.353441 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.353369 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdce05b3-643e-4901-b9dd-027049a2a4c7-kube-api-access-jplkc" pod "fdce05b3-643e-4901-b9dd-027049a2a4c7" (UID: "fdce05b3-643e-4901-b9dd-027049a2a4c7"). InnerVolumeSpecName "kube-api-access-jplkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:45.454283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.454249 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jplkc\" (UniqueName: \"kubernetes.io/projected/fdce05b3-643e-4901-b9dd-027049a2a4c7-kube-api-access-jplkc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:45.454283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.454290 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdce05b3-643e-4901-b9dd-027049a2a4c7-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:45.649531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.649446 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:45.659742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.659706 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 01 23:00:45.659884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:45.659769 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 01 23:00:46.020773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.020746 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" Nov 01 23:00:46.020773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.020757 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs" event={"ID":"fdce05b3-643e-4901-b9dd-027049a2a4c7","Type":"ContainerDied","Data":"51830f0f54273d16e78b70e7236a289973a50b4e12e2b83dab44ae5e0477866f"} Nov 01 23:00:46.021049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.020801 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51830f0f54273d16e78b70e7236a289973a50b4e12e2b83dab44ae5e0477866f" Nov 01 23:00:46.205333 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.205297 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:46.386651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.379458 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:46.660118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.660044 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 01 23:00:46.660118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.660098 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 01 23:00:46.728660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.728619 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:46.744154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.744116 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" path="/var/lib/kubelet/pods/c3c9e058-877f-41e0-8e59-ac5b17f987fc/volumes" Nov 01 23:00:46.744512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.744495 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v"] Nov 01 23:00:46.744659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.744527 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f675d-b2h6v"] Nov 01 23:00:46.828204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:46.828169 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:47.249783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.242315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:47.605123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.605089 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 23:00:47.613651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.613585 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 23:00:47.615740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.615714 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 23:00:47.651972 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.651944 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:47.660539 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.660516 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 23:00:47.662866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.662845 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 23:00:47.668724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.668700 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:47.670060 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.670038 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:47.673739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.673716 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:00:47.682518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.682497 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 23:00:47.684648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.684631 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710629 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istiod-ca-cert\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710686 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-credential-socket\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710720 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-kube-api-access-f84x8\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-kube-api-access-f84x8\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710752 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-podinfo\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710791 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istiod-ca-cert\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710820 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-kube-api-access-tqvc2\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-kube-api-access-tqvc2\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710868 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-data\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710905 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-podinfo\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.710940 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-podinfo\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711001 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-socket\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711029 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-data\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711122 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-token\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-token\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711325 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-token\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-token\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711359 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-certs\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711415 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7c2c33-56f7-45c5-911d-0413c08efb63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c2c33-56f7-45c5-911d-0413c08efb63-istiod-ca-cert\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711494 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-credential-socket\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711558 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-envoy\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711602 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51812fed-7df5-42f0-9e69-70130fe330bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51812fed-7df5-42f0-9e69-70130fe330bd-istio-podinfo\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.711636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711634 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58874768-db63-47ff-b564-ff6c26f84b86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58874768-db63-47ff-b564-ff6c26f84b86-istio-podinfo\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711683 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-certs\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711717 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-envoy\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711745 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-kube-api-access-hcgst\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-kube-api-access-hcgst\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711783 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-socket\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711832 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-socket\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711888 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a51428-3f39-46e6-8057-6518fce34f4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a51428-3f39-46e6-8057-6518fce34f4d-istiod-ca-cert\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711944 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-token\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-token\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.711990 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-envoy\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712038 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-certs\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712112 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-kube-api-access-q2tqm\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-kube-api-access-q2tqm\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712200 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-podinfo\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712245 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-credential-socket\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712305 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-data\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712350 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-envoy\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712403 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-certs\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712435 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-certs\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712466 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-istio-token\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-istio-token\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712501 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-token\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-token\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712530 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-envoy\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712566 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a708cdd3-2466-4846-876a-99b7f2c35bfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a708cdd3-2466-4846-876a-99b7f2c35bfc-istiod-ca-cert\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712624 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-envoy\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712682 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-certs\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712842 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-token\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.712912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712878 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-data\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.712976 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-istio-token\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-istio-token\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713010 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-kube-api-access-gh97q\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-kube-api-access-gh97q\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713069 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-credential-socket\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713116 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-socket\") pod \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\" (UID: \"a949c1c2-3db6-427c-a5cf-29824eb9d3d6\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713148 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-data\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713176 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-credential-socket\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713209 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-socket\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713239 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-data\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713287 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-certs\") pod \"a708cdd3-2466-4846-876a-99b7f2c35bfc\" (UID: \"a708cdd3-2466-4846-876a-99b7f2c35bfc\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713348 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-envoy\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713380 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-kube-api-access-lh77z\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-kube-api-access-lh77z\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713422 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-credential-socket\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713486 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-data\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713534 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-podinfo\") pod \"d1a51428-3f39-46e6-8057-6518fce34f4d\" (UID: \"d1a51428-3f39-46e6-8057-6518fce34f4d\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713614 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-kube-api-access-dfbj8\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-kube-api-access-dfbj8\") pod \"fc7c2c33-56f7-45c5-911d-0413c08efb63\" (UID: \"fc7c2c33-56f7-45c5-911d-0413c08efb63\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713689 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-socket\") pod \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\" (UID: \"b7f690d3-a45f-4ebf-b366-24bd677fe19e\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713717 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-credential-socket\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713747 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58874768-db63-47ff-b564-ff6c26f84b86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58874768-db63-47ff-b564-ff6c26f84b86-istiod-ca-cert\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713801 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-socket\") pod \"58874768-db63-47ff-b564-ff6c26f84b86\" (UID: \"58874768-db63-47ff-b564-ff6c26f84b86\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.713840 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51812fed-7df5-42f0-9e69-70130fe330bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51812fed-7df5-42f0-9e69-70130fe330bd-istiod-ca-cert\") pod \"51812fed-7df5-42f0-9e69-70130fe330bd\" (UID: \"51812fed-7df5-42f0-9e69-70130fe330bd\") " Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.716318 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istiod-ca-cert" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.716625 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-credential-socket" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.723968 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-podinfo" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.740837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.727841 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istiod-ca-cert" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.747784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.747732 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-data" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.751369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.751337 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-envoy" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.751556 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.751366 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-podinfo" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.754653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.754565 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-envoy" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.759927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.759746 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-credential-socket" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.761337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.761239 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-data" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.762889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.762860 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a708cdd3-2466-4846-876a-99b7f2c35bfc-istiod-ca-cert" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.763144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.763124 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-envoy" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.763252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.763134 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-certs" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.763370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.763354 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-credential-socket" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.763454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.763405 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-credential-socket" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.763676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.763653 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-data" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.763924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.763905 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-socket" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.764024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.763945 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51812fed-7df5-42f0-9e69-70130fe330bd-istiod-ca-cert" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.764972 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.764710 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-kube-api-access-f84x8" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "kube-api-access-f84x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.765301 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-socket" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.765484 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-data" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.765662 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-data" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.765818 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-socket" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.766009 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-socket" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.768921 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-kube-api-access-dfbj8" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "kube-api-access-dfbj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.769145 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-socket" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.769333 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-socket" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.769502 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-data" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.770891 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-certs" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.771075 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-credential-socket" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.774172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.773924 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-socket" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.775096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.774302 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a51428-3f39-46e6-8057-6518fce34f4d-istiod-ca-cert" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.775096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.774879 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-certs" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.775968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.775939 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-credential-socket" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.777993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.777968 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-certs" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.782232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.781014 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-certs" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.782232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.781826 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-data" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.782232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.782005 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-token" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.782783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.782734 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-certs" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.784022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.783054 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58874768-db63-47ff-b564-ff6c26f84b86-istiod-ca-cert" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.784022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.783343 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7c2c33-56f7-45c5-911d-0413c08efb63-istiod-ca-cert" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:47.784022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.783427 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-podinfo" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.784022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.783682 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-credential-socket" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.786243 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.784900 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-kube-api-access-q2tqm" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "kube-api-access-q2tqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.802246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.786533 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-certs" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.802246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.794147 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51812fed-7df5-42f0-9e69-70130fe330bd-istio-podinfo" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.802246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.794171 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-token" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.802246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.798248 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-token" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.802246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.798316 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-kube-api-access-tqvc2" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "kube-api-access-tqvc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.805637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.804297 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-kube-api-access-hcgst" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "kube-api-access-hcgst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.808238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.808190 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-envoy" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.808238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.808223 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-istio-token" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.809934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.809896 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-envoy" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.809934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.809922 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-podinfo" pod "a708cdd3-2466-4846-876a-99b7f2c35bfc" (UID: "a708cdd3-2466-4846-876a-99b7f2c35bfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.810109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.809954 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-envoy" pod "a949c1c2-3db6-427c-a5cf-29824eb9d3d6" (UID: "a949c1c2-3db6-427c-a5cf-29824eb9d3d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.810109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.810067 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-istio-token" pod "51812fed-7df5-42f0-9e69-70130fe330bd" (UID: "51812fed-7df5-42f0-9e69-70130fe330bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.810366 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.810329 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-podinfo" pod "d1a51428-3f39-46e6-8057-6518fce34f4d" (UID: "d1a51428-3f39-46e6-8057-6518fce34f4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.810366 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.810358 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-kube-api-access-gh97q" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "kube-api-access-gh97q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.810597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.810378 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-token" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.812125 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.812090 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-envoy" pod "fc7c2c33-56f7-45c5-911d-0413c08efb63" (UID: "fc7c2c33-56f7-45c5-911d-0413c08efb63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:00:47.812353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.812335 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-kube-api-access-lh77z" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "kube-api-access-lh77z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.813025 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.812992 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-token" pod "b7f690d3-a45f-4ebf-b366-24bd677fe19e" (UID: "b7f690d3-a45f-4ebf-b366-24bd677fe19e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:47.813641 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.813436 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58874768-db63-47ff-b564-ff6c26f84b86-istio-podinfo" pod "58874768-db63-47ff-b564-ff6c26f84b86" (UID: "58874768-db63-47ff-b564-ff6c26f84b86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817622 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817654 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817664 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c2c33-56f7-45c5-911d-0413c08efb63-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817674 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817683 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817692 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51812fed-7df5-42f0-9e69-70130fe330bd-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817702 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58874768-db63-47ff-b564-ff6c26f84b86-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817711 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817719 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817728 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgst\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-kube-api-access-hcgst\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817739 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817747 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817755 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a51428-3f39-46e6-8057-6518fce34f4d-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817765 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817773 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817782 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817795 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2tqm\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-kube-api-access-q2tqm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817807 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817816 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817826 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817834 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817844 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817860 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817868 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817876 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817885 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817893 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a708cdd3-2466-4846-876a-99b7f2c35bfc-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817901 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817911 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817919 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817928 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817935 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817946 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh97q\" (UniqueName: \"kubernetes.io/projected/58874768-db63-47ff-b564-ff6c26f84b86-kube-api-access-gh97q\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817955 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817964 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817973 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817981 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817990 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.817997 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818006 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818014 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818022 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh77z\" (UniqueName: \"kubernetes.io/projected/b7f690d3-a45f-4ebf-b366-24bd677fe19e-kube-api-access-lh77z\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818030 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818039 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818047 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818055 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbj8\" (UniqueName: \"kubernetes.io/projected/fc7c2c33-56f7-45c5-911d-0413c08efb63-kube-api-access-dfbj8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818064 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7f690d3-a45f-4ebf-b366-24bd677fe19e-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818073 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51812fed-7df5-42f0-9e69-70130fe330bd-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818081 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58874768-db63-47ff-b564-ff6c26f84b86-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818089 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58874768-db63-47ff-b564-ff6c26f84b86-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818098 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51812fed-7df5-42f0-9e69-70130fe330bd-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818106 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818116 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c2c33-56f7-45c5-911d-0413c08efb63-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818125 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84x8\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-kube-api-access-f84x8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818134 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818142 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a949c1c2-3db6-427c-a5cf-29824eb9d3d6-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818150 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqvc2\" (UniqueName: \"kubernetes.io/projected/51812fed-7df5-42f0-9e69-70130fe330bd-kube-api-access-tqvc2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818159 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a51428-3f39-46e6-8057-6518fce34f4d-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818168 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7f690d3-a45f-4ebf-b366-24bd677fe19e-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818177 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c2c33-56f7-45c5-911d-0413c08efb63-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818185 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818193 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:47.818219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:47.818202 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a708cdd3-2466-4846-876a-99b7f2c35bfc-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:48.059322 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.059235 4981 generic.go:356] "Generic (PLEG): container finished" podID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerID="053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871" exitCode=0 Nov 01 23:00:48.059322 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.059304 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" Nov 01 23:00:48.059563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.059320 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" event={"ID":"d1a51428-3f39-46e6-8057-6518fce34f4d","Type":"ContainerDied","Data":"053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871"} Nov 01 23:00:48.059563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.059384 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n" event={"ID":"d1a51428-3f39-46e6-8057-6518fce34f4d","Type":"ContainerDied","Data":"cb7057feebd804560227059aa8ce7f510e2fd6531f2f95650a89306ae3c60954"} Nov 01 23:00:48.059563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.059449 4981 scope.go:122] "RemoveContainer" containerID="053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871" Nov 01 23:00:48.061122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.061100 4981 generic.go:356] "Generic (PLEG): container finished" podID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerID="d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957" exitCode=0 Nov 01 23:00:48.061249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.061176 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" Nov 01 23:00:48.061249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.061191 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" event={"ID":"a708cdd3-2466-4846-876a-99b7f2c35bfc","Type":"ContainerDied","Data":"d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957"} Nov 01 23:00:48.061249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.061234 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb" event={"ID":"a708cdd3-2466-4846-876a-99b7f2c35bfc","Type":"ContainerDied","Data":"d81023c7b86eb40bf1bb785f28d1db8a70fc967e7356c14c44ec0982b0aeaeff"} Nov 01 23:00:48.064564 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.064543 4981 generic.go:356] "Generic (PLEG): container finished" podID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerID="3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec" exitCode=0 Nov 01 23:00:48.064785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.064604 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" Nov 01 23:00:48.064785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.064601 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" event={"ID":"a949c1c2-3db6-427c-a5cf-29824eb9d3d6","Type":"ContainerDied","Data":"3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec"} Nov 01 23:00:48.064912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.064795 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577" event={"ID":"a949c1c2-3db6-427c-a5cf-29824eb9d3d6","Type":"ContainerDied","Data":"d81f877049779f0935fc0d85a00ccd4d59290cf6ee8162fca8f2c8d5946fa3a7"} Nov 01 23:00:48.069706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.066290 4981 generic.go:356] "Generic (PLEG): container finished" podID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerID="21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6" exitCode=0 Nov 01 23:00:48.069706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.066405 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" event={"ID":"fc7c2c33-56f7-45c5-911d-0413c08efb63","Type":"ContainerDied","Data":"21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6"} Nov 01 23:00:48.069706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.066443 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" event={"ID":"fc7c2c33-56f7-45c5-911d-0413c08efb63","Type":"ContainerDied","Data":"b396ae9fc6dc65ecd665acabe40e02e78f1eb8fff86260e40f20619614d68384"} Nov 01 23:00:48.069706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.066503 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn" Nov 01 23:00:48.071651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.071627 4981 generic.go:356] "Generic (PLEG): container finished" podID="58874768-db63-47ff-b564-ff6c26f84b86" containerID="b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0" exitCode=0 Nov 01 23:00:48.071766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.071751 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" event={"ID":"58874768-db63-47ff-b564-ff6c26f84b86","Type":"ContainerDied","Data":"b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0"} Nov 01 23:00:48.071837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.071788 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" event={"ID":"58874768-db63-47ff-b564-ff6c26f84b86","Type":"ContainerDied","Data":"2826c001011c69478b012116842e5dcfd759ef97f2cac8eb557173540c3d4aa1"} Nov 01 23:00:48.071900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.071841 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd" Nov 01 23:00:48.073868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.073820 4981 generic.go:356] "Generic (PLEG): container finished" podID="51812fed-7df5-42f0-9e69-70130fe330bd" containerID="96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025" exitCode=0 Nov 01 23:00:48.073868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.073844 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" event={"ID":"51812fed-7df5-42f0-9e69-70130fe330bd","Type":"ContainerDied","Data":"96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025"} Nov 01 23:00:48.074040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.073902 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" event={"ID":"51812fed-7df5-42f0-9e69-70130fe330bd","Type":"ContainerDied","Data":"b733aa8808bef1a72a92534ed40519566127635084dd28babd78612da63f514c"} Nov 01 23:00:48.074165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.074141 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4" Nov 01 23:00:48.078496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.078476 4981 generic.go:356] "Generic (PLEG): container finished" podID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerID="f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1" exitCode=0 Nov 01 23:00:48.078892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.078509 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" event={"ID":"b7f690d3-a45f-4ebf-b366-24bd677fe19e","Type":"ContainerDied","Data":"f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1"} Nov 01 23:00:48.079032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.078988 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" event={"ID":"b7f690d3-a45f-4ebf-b366-24bd677fe19e","Type":"ContainerDied","Data":"33fdc08c3b66744dca287926ca06645218e80da39448722c23349c2d3c1b2872"} Nov 01 23:00:48.079032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.078541 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl" Nov 01 23:00:48.079599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.079396 4981 scope.go:122] "RemoveContainer" containerID="053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871" Nov 01 23:00:48.079750 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:48.079725 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871\": container with ID starting with 053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871 not found: ID does not exist" containerID="053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871" Nov 01 23:00:48.079941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.079878 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871"} err="failed to get container status \"053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871\": rpc error: code = NotFound desc = could not find container \"053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871\": container with ID starting with 053a4008d6ea0769359fd5225e87d8a3fc6764322104724107f99aeb8eb59871 not found: ID does not exist" Nov 01 23:00:48.079941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.079909 4981 scope.go:122] "RemoveContainer" containerID="d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957" Nov 01 23:00:48.096520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.096501 4981 scope.go:122] "RemoveContainer" containerID="d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957" Nov 01 23:00:48.096899 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:48.096877 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957\": container with ID starting with d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957 not found: ID does not exist" containerID="d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957" Nov 01 23:00:48.096994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.096903 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957"} err="failed to get container status \"d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957\": rpc error: code = NotFound desc = could not find container \"d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957\": container with ID starting with d42f47ce61a711947ff70805cddcff62529a699dae5bd7fb47dfa623eabe6957 not found: ID does not exist" Nov 01 23:00:48.096994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.096920 4981 scope.go:122] "RemoveContainer" containerID="3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec" Nov 01 23:00:48.099843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.099821 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n"] Nov 01 23:00:48.105397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.105377 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-z5r4n"] Nov 01 23:00:48.108989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.108972 4981 scope.go:122] "RemoveContainer" containerID="3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec" Nov 01 23:00:48.109299 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:48.109279 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec\": container with ID starting with 3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec not found: ID does not exist" containerID="3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec" Nov 01 23:00:48.109349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.109307 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec"} err="failed to get container status \"3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec\": rpc error: code = NotFound desc = could not find container \"3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec\": container with ID starting with 3b9f099bd0c4cee32fd076532ade204fff8805987e6d556f9772ed58b4af8aec not found: ID does not exist" Nov 01 23:00:48.109349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.109324 4981 scope.go:122] "RemoveContainer" containerID="21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6" Nov 01 23:00:48.119233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.116533 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb"] Nov 01 23:00:48.127193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.127170 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-ptdwb"] Nov 01 23:00:48.128891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.128873 4981 scope.go:122] "RemoveContainer" containerID="21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6" Nov 01 23:00:48.129197 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:48.129176 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6\": container with ID starting with 21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6 not found: ID does not exist" containerID="21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6" Nov 01 23:00:48.129278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.129203 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6"} err="failed to get container status \"21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6\": rpc error: code = NotFound desc = could not find container \"21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6\": container with ID starting with 21d03fda4ea2746b07f9e37df0e3bd1cdee1ff3b79afa4c28baa281f52e330d6 not found: ID does not exist" Nov 01 23:00:48.129278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.129221 4981 scope.go:122] "RemoveContainer" containerID="b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0" Nov 01 23:00:48.139326 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.139303 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577"] Nov 01 23:00:48.142991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.142975 4981 scope.go:122] "RemoveContainer" containerID="b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0" Nov 01 23:00:48.143286 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:48.143261 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0\": container with ID starting with b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0 not found: ID does not exist" containerID="b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0" Nov 01 23:00:48.143343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.143297 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0"} err="failed to get container status \"b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0\": rpc error: code = NotFound desc = could not find container \"b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0\": container with ID starting with b5ac010a595d5b9561f4671969bd25350c3fedcb3a6f52ac1784820d7b7a66e0 not found: ID does not exist" Nov 01 23:00:48.143343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.143323 4981 scope.go:122] "RemoveContainer" containerID="96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025" Nov 01 23:00:48.144192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.144176 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-cw577"] Nov 01 23:00:48.153471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.153444 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd"] Nov 01 23:00:48.154708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.154689 4981 scope.go:122] "RemoveContainer" containerID="96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025" Nov 01 23:00:48.154965 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:48.154940 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025\": container with ID starting with 96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025 not found: ID does not exist" containerID="96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025" Nov 01 23:00:48.155016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.154976 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025"} err="failed to get container status \"96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025\": rpc error: code = NotFound desc = could not find container \"96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025\": container with ID starting with 96b7b7287ca0dcd10d5fd55fe6ff84bdd00f0acf9431a95c1f05dae43e797025 not found: ID does not exist" Nov 01 23:00:48.155016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.154999 4981 scope.go:122] "RemoveContainer" containerID="f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1" Nov 01 23:00:48.160952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.160931 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-jskwd"] Nov 01 23:00:48.167743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.167721 4981 scope.go:122] "RemoveContainer" containerID="f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1" Nov 01 23:00:48.168982 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:48.168955 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1\": container with ID starting with f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1 not found: ID does not exist" containerID="f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1" Nov 01 23:00:48.169091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.168987 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1"} err="failed to get container status \"f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1\": rpc error: code = NotFound desc = could not find container \"f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1\": container with ID starting with f456de48e6608fb47a2c394ea6210b2a62b7ff299e9961c78544a7dc65f5abb1 not found: ID does not exist" Nov 01 23:00:48.173460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.173437 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl"] Nov 01 23:00:48.180799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.180778 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-zm7sl"] Nov 01 23:00:48.194823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.194795 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn"] Nov 01 23:00:48.200110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.200087 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-66hsn"] Nov 01 23:00:48.212740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.212710 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4"] Nov 01 23:00:48.216569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.216552 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-021714-openshift-default-5f7f79655c-rclq4"] Nov 01 23:00:48.596131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.596096 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:48.737397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.737362 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" path="/var/lib/kubelet/pods/51812fed-7df5-42f0-9e69-70130fe330bd/volumes" Nov 01 23:00:48.738163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.738140 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58874768-db63-47ff-b564-ff6c26f84b86" path="/var/lib/kubelet/pods/58874768-db63-47ff-b564-ff6c26f84b86/volumes" Nov 01 23:00:48.738795 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.738776 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ded17e7-24f9-46e9-94a4-ec1f75aa2aee" path="/var/lib/kubelet/pods/8ded17e7-24f9-46e9-94a4-ec1f75aa2aee/volumes" Nov 01 23:00:48.740073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.740048 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" path="/var/lib/kubelet/pods/a708cdd3-2466-4846-876a-99b7f2c35bfc/volumes" Nov 01 23:00:48.740817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.740799 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" path="/var/lib/kubelet/pods/a949c1c2-3db6-427c-a5cf-29824eb9d3d6/volumes" Nov 01 23:00:48.741484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.741466 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" path="/var/lib/kubelet/pods/b7f690d3-a45f-4ebf-b366-24bd677fe19e/volumes" Nov 01 23:00:48.742561 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.742542 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" path="/var/lib/kubelet/pods/d1a51428-3f39-46e6-8057-6518fce34f4d/volumes" Nov 01 23:00:48.743224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:48.743205 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" path="/var/lib/kubelet/pods/fc7c2c33-56f7-45c5-911d-0413c08efb63/volumes" Nov 01 23:00:49.396802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:49.396764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:50.304009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:50.303979 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:51.610976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:51.610941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:52.096139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:52.095540 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g"] Nov 01 23:00:52.238178 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:52.238147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:52.341686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:52.341647 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:52.753146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:52.753117 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:53.104332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.104234 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:53.205557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.205523 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:53.365671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.359570 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" podUID="369ead52-8316-4afd-9b8f-ab992cd76b7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:53.400663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.400629 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:53.794302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.794271 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:53.882410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.882357 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572"] Nov 01 23:00:53.883813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883786 4981 cpu_manager.go:414] "containerMap: removing container" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" Nov 01 23:00:53.883813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883815 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883834 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883842 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883871 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883881 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883907 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerName="payload-pre-processing" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883914 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerName="payload-pre-processing" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883935 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883943 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883971 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fdce05b3-643e-4901-b9dd-027049a2a4c7" containerName="revoke-keys" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883978 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdce05b3-643e-4901-b9dd-027049a2a4c7" containerName="revoke-keys" Nov 01 23:00:53.883989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883991 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" Nov 01 23:00:53.884566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.883999 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" Nov 01 23:00:53.884566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.884012 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" Nov 01 23:00:53.884566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.884019 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" Nov 01 23:00:53.884566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.884047 4981 cpu_manager.go:414] "containerMap: removing container" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" Nov 01 23:00:53.884566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.884055 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" Nov 01 23:00:53.884566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.884229 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a51428-3f39-46e6-8057-6518fce34f4d" containerName="istio-proxy" Nov 01 23:00:53.892839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892810 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a949c1c2-3db6-427c-a5cf-29824eb9d3d6" containerName="istio-proxy" Nov 01 23:00:53.893006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892853 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c9e058-877f-41e0-8e59-ac5b17f987fc" containerName="payload-pre-processing" Nov 01 23:00:53.893006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892869 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a708cdd3-2466-4846-876a-99b7f2c35bfc" containerName="istio-proxy" Nov 01 23:00:53.893006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892887 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdce05b3-643e-4901-b9dd-027049a2a4c7" containerName="revoke-keys" Nov 01 23:00:53.893006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892899 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7c2c33-56f7-45c5-911d-0413c08efb63" containerName="istio-proxy" Nov 01 23:00:53.893006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892910 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f690d3-a45f-4ebf-b366-24bd677fe19e" containerName="istio-proxy" Nov 01 23:00:53.893006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892929 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="51812fed-7df5-42f0-9e69-70130fe330bd" containerName="istio-proxy" Nov 01 23:00:53.893006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.892944 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="58874768-db63-47ff-b564-ff6c26f84b86" containerName="istio-proxy" Nov 01 23:00:53.900907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.899789 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572"] Nov 01 23:00:53.900907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.899972 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:53.907420 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.907389 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:00:53.907616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.907525 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:00:53.984596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.984543 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:53.984771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.984741 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:53.984889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:53.984867 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mskv\" (UniqueName: \"kubernetes.io/projected/6761e04f-af57-463b-bf6f-c1a554cabe74-kube-api-access-6mskv\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.086634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.086518 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mskv\" (UniqueName: \"kubernetes.io/projected/6761e04f-af57-463b-bf6f-c1a554cabe74-kube-api-access-6mskv\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.086801 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.086721 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.086801 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.086793 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.087399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.087369 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.089194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.089168 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.098842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.098820 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mskv\" (UniqueName: \"kubernetes.io/projected/6761e04f-af57-463b-bf6f-c1a554cabe74-kube-api-access-6mskv\") pod \"maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.232124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.232092 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:54.397891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.397796 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" podUID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:00:54.522902 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:00:54.522873 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6761e04f_af57_463b_bf6f_c1a554cabe74.slice/crio-37d8214f18194653ec63ae8084ecd2fb3d8101774942101b6d7f1e85f0c3b9c6 WatchSource:0}: Error finding container 37d8214f18194653ec63ae8084ecd2fb3d8101774942101b6d7f1e85f0c3b9c6: Status 404 returned error can't find the container with id 37d8214f18194653ec63ae8084ecd2fb3d8101774942101b6d7f1e85f0c3b9c6 Nov 01 23:00:54.523758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:54.523731 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572"] Nov 01 23:00:55.041383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:55.041299 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:55.204785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:55.204744 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" event={"ID":"6761e04f-af57-463b-bf6f-c1a554cabe74","Type":"ContainerStarted","Data":"1e39ef2c93737dd469452bcacbfa4eeb203f428ee3d65a9789adea1cffcd6dc3"} Nov 01 23:00:55.204965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:55.204813 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" event={"ID":"6761e04f-af57-463b-bf6f-c1a554cabe74","Type":"ContainerStarted","Data":"37d8214f18194653ec63ae8084ecd2fb3d8101774942101b6d7f1e85f0c3b9c6"} Nov 01 23:00:55.241481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:55.241431 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" podStartSLOduration=2.241415548 podStartE2EDuration="2.241415548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:55.2275607 +0000 UTC m=+2220.655438634" watchObservedRunningTime="2026-11-01 23:00:55.241415548 +0000 UTC m=+2220.669293479" Nov 01 23:00:55.775198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:55.775121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:56.215475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.215435 4981 generic.go:356] "Generic (PLEG): container finished" podID="6761e04f-af57-463b-bf6f-c1a554cabe74" containerID="1e39ef2c93737dd469452bcacbfa4eeb203f428ee3d65a9789adea1cffcd6dc3" exitCode=0 Nov 01 23:00:56.215969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.215480 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" event={"ID":"6761e04f-af57-463b-bf6f-c1a554cabe74","Type":"ContainerDied","Data":"1e39ef2c93737dd469452bcacbfa4eeb203f428ee3d65a9789adea1cffcd6dc3"} Nov 01 23:00:56.461028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.460997 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:56.852517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.852491 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 23:00:56.856249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.856229 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 23:00:56.936899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.936866 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/369ead52-8316-4afd-9b8f-ab992cd76b7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369ead52-8316-4afd-9b8f-ab992cd76b7d-plugins-config-volume\") pod \"369ead52-8316-4afd-9b8f-ab992cd76b7d\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " Nov 01 23:00:56.937056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.936993 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e21a98-7a1e-45d7-a97f-8344eefe7843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e21a98-7a1e-45d7-a97f-8344eefe7843-plugins-config-volume\") pod \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " Nov 01 23:00:56.937056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.937031 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/369ead52-8316-4afd-9b8f-ab992cd76b7d-kube-api-access-8fgmt\" (UniqueName: \"kubernetes.io/projected/369ead52-8316-4afd-9b8f-ab992cd76b7d-kube-api-access-8fgmt\") pod \"369ead52-8316-4afd-9b8f-ab992cd76b7d\" (UID: \"369ead52-8316-4afd-9b8f-ab992cd76b7d\") " Nov 01 23:00:56.937159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.937124 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e21a98-7a1e-45d7-a97f-8344eefe7843-kube-api-access-nnxh9\" (UniqueName: \"kubernetes.io/projected/e1e21a98-7a1e-45d7-a97f-8344eefe7843-kube-api-access-nnxh9\") pod \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\" (UID: \"e1e21a98-7a1e-45d7-a97f-8344eefe7843\") " Nov 01 23:00:56.937330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.937308 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369ead52-8316-4afd-9b8f-ab992cd76b7d-plugins-config-volume" pod "369ead52-8316-4afd-9b8f-ab992cd76b7d" (UID: "369ead52-8316-4afd-9b8f-ab992cd76b7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:56.937790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.937760 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e21a98-7a1e-45d7-a97f-8344eefe7843-plugins-config-volume" pod "e1e21a98-7a1e-45d7-a97f-8344eefe7843" (UID: "e1e21a98-7a1e-45d7-a97f-8344eefe7843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:56.937955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.937932 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/369ead52-8316-4afd-9b8f-ab992cd76b7d-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:56.937955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.937957 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e21a98-7a1e-45d7-a97f-8344eefe7843-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:56.939924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.939902 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e21a98-7a1e-45d7-a97f-8344eefe7843-kube-api-access-nnxh9" pod "e1e21a98-7a1e-45d7-a97f-8344eefe7843" (UID: "e1e21a98-7a1e-45d7-a97f-8344eefe7843"). InnerVolumeSpecName "kube-api-access-nnxh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:56.940004 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:56.939937 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369ead52-8316-4afd-9b8f-ab992cd76b7d-kube-api-access-8fgmt" pod "369ead52-8316-4afd-9b8f-ab992cd76b7d" (UID: "369ead52-8316-4afd-9b8f-ab992cd76b7d"). InnerVolumeSpecName "kube-api-access-8fgmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:57.039799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.039717 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxh9\" (UniqueName: \"kubernetes.io/projected/e1e21a98-7a1e-45d7-a97f-8344eefe7843-kube-api-access-nnxh9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:57.039799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.039749 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgmt\" (UniqueName: \"kubernetes.io/projected/369ead52-8316-4afd-9b8f-ab992cd76b7d-kube-api-access-8fgmt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:57.096833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.096772 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" podUID="fdfcc93a-a688-497a-aabf-801bd24d2159" containerName="payload-pre-processing" containerID="cri-o://bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264" gracePeriod=25 Nov 01 23:00:57.145978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.145945 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:57.265187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.265142 4981 generic.go:356] "Generic (PLEG): container finished" podID="369ead52-8316-4afd-9b8f-ab992cd76b7d" containerID="aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65" exitCode=0 Nov 01 23:00:57.265684 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.265193 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" Nov 01 23:00:57.265866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.265821 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" event={"ID":"369ead52-8316-4afd-9b8f-ab992cd76b7d","Type":"ContainerDied","Data":"aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65"} Nov 01 23:00:57.266030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.266013 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft" event={"ID":"369ead52-8316-4afd-9b8f-ab992cd76b7d","Type":"ContainerDied","Data":"cdaf679ddae454621f47f02d9d33a7a5993ec57e675fdd772baf86380aac1f8c"} Nov 01 23:00:57.266143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.266060 4981 scope.go:122] "RemoveContainer" containerID="aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65" Nov 01 23:00:57.268073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.268045 4981 generic.go:356] "Generic (PLEG): container finished" podID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" containerID="eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546" exitCode=0 Nov 01 23:00:57.268176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.268079 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" Nov 01 23:00:57.268273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.268248 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" event={"ID":"e1e21a98-7a1e-45d7-a97f-8344eefe7843","Type":"ContainerDied","Data":"eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546"} Nov 01 23:00:57.268394 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.268294 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb" event={"ID":"e1e21a98-7a1e-45d7-a97f-8344eefe7843","Type":"ContainerDied","Data":"5d5d1eb7c72da78345b92547d87b197f4b4cec0f1e1b818edfea8a274e80b9ce"} Nov 01 23:00:57.281446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.281428 4981 scope.go:122] "RemoveContainer" containerID="aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65" Nov 01 23:00:57.281809 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:57.281786 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65\": container with ID starting with aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65 not found: ID does not exist" containerID="aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65" Nov 01 23:00:57.281907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.281818 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65"} err="failed to get container status \"aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65\": rpc error: code = NotFound desc = could not find container \"aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65\": container with ID starting with aba392d565cc0abd6ac992fe2e161212d022340a7306d1d85a2691e6cecf3f65 not found: ID does not exist" Nov 01 23:00:57.281907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.281834 4981 scope.go:122] "RemoveContainer" containerID="eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546" Nov 01 23:00:57.294025 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.293945 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft"] Nov 01 23:00:57.298838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.298803 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2f675d-6745758bb7-h6nft"] Nov 01 23:00:57.300033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.299936 4981 scope.go:122] "RemoveContainer" containerID="eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546" Nov 01 23:00:57.300645 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:00:57.300271 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546\": container with ID starting with eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546 not found: ID does not exist" containerID="eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546" Nov 01 23:00:57.300645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.300305 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546"} err="failed to get container status \"eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546\": rpc error: code = NotFound desc = could not find container \"eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546\": container with ID starting with eb6b95c3490686d6a05852d1ec97718e521f8ee9028470e47a31b86279129546 not found: ID does not exist" Nov 01 23:00:57.315363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.315338 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb"] Nov 01 23:00:57.319904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.319878 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2f675d-5d84d8778f-r6lqb"] Nov 01 23:00:57.422663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.422636 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:57.550507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.550425 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6761e04f-af57-463b-bf6f-c1a554cabe74-kube-api-access-6mskv\" (UniqueName: \"kubernetes.io/projected/6761e04f-af57-463b-bf6f-c1a554cabe74-kube-api-access-6mskv\") pod \"6761e04f-af57-463b-bf6f-c1a554cabe74\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " Nov 01 23:00:57.550507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.550485 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-xks-ca\") pod \"6761e04f-af57-463b-bf6f-c1a554cabe74\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " Nov 01 23:00:57.550717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.550527 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-openshift-ca\") pod \"6761e04f-af57-463b-bf6f-c1a554cabe74\" (UID: \"6761e04f-af57-463b-bf6f-c1a554cabe74\") " Nov 01 23:00:57.550980 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.550958 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-openshift-ca" pod "6761e04f-af57-463b-bf6f-c1a554cabe74" (UID: "6761e04f-af57-463b-bf6f-c1a554cabe74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:00:57.551211 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.551192 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:57.552612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.552564 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-xks-ca" pod "6761e04f-af57-463b-bf6f-c1a554cabe74" (UID: "6761e04f-af57-463b-bf6f-c1a554cabe74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:00:57.552733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.552664 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6761e04f-af57-463b-bf6f-c1a554cabe74-kube-api-access-6mskv" pod "6761e04f-af57-463b-bf6f-c1a554cabe74" (UID: "6761e04f-af57-463b-bf6f-c1a554cabe74"). InnerVolumeSpecName "kube-api-access-6mskv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:00:57.652689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.652654 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mskv\" (UniqueName: \"kubernetes.io/projected/6761e04f-af57-463b-bf6f-c1a554cabe74-kube-api-access-6mskv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:57.652689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.652685 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6761e04f-af57-463b-bf6f-c1a554cabe74-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:00:57.951214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:57.951181 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:58.277871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:58.277788 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" Nov 01 23:00:58.277871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:58.277823 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572" event={"ID":"6761e04f-af57-463b-bf6f-c1a554cabe74","Type":"ContainerDied","Data":"37d8214f18194653ec63ae8084ecd2fb3d8101774942101b6d7f1e85f0c3b9c6"} Nov 01 23:00:58.277871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:58.277872 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37d8214f18194653ec63ae8084ecd2fb3d8101774942101b6d7f1e85f0c3b9c6" Nov 01 23:00:58.576568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:58.576483 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:00:58.737711 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:58.737678 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="369ead52-8316-4afd-9b8f-ab992cd76b7d" path="/var/lib/kubelet/pods/369ead52-8316-4afd-9b8f-ab992cd76b7d/volumes" Nov 01 23:00:58.738277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:58.738259 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" path="/var/lib/kubelet/pods/e1e21a98-7a1e-45d7-a97f-8344eefe7843/volumes" Nov 01 23:00:59.298514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:00:59.298479 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:00.017532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:00.017498 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:00.918195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:00.918164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:01.463700 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:01.463667 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:01.589569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:01.589543 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:01.613184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:01.613154 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs"] Nov 01 23:01:01.618162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:01.618093 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39e952-kx4qs"] Nov 01 23:01:01.690983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:01.690935 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:02.069804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:02.069766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:02.640440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:02.640401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:02.739508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:02.739473 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdce05b3-643e-4901-b9dd-027049a2a4c7" path="/var/lib/kubelet/pods/fdce05b3-643e-4901-b9dd-027049a2a4c7/volumes" Nov 01 23:01:02.777612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:02.777540 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" podUID="fdfcc93a-a688-497a-aabf-801bd24d2159" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:01:03.447117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:03.447084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:03.989284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:03.989253 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5"] Nov 01 23:01:04.005404 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:04.005371 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv"] Nov 01 23:01:04.066464 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:04.066426 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:04.161975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:04.161941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:04.298663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:04.296386 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:04.947334 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:04.947297 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:05.546143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:05.546113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:06.299748 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:06.299704 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:07.138475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:07.138447 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:07.864500 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:07.864467 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:08.220692 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:08.220664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:08.319618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:08.319549 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:08.511677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:08.507126 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv"] Nov 01 23:01:08.511677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:08.507459 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" containerID="cri-o://0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803" gracePeriod=30 Nov 01 23:01:08.540083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:08.540048 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:08.826401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:08.826316 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:08.990431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:08.990389 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" podUID="ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" containerName="payload-processing" containerID="cri-o://42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a" gracePeriod=25 Nov 01 23:01:09.005951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.005912 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" podUID="def232dc-e82c-4e0c-851a-bca840f18505" containerName="payload-pre-processing" containerID="cri-o://5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1" gracePeriod=25 Nov 01 23:01:09.284919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.284881 4981 patch_prober.go:29] interesting pod/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:01:09.285033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.284955 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:01:09.303067 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.303038 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 23:01:09.368757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.368685 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-kube-api-access-zrwkx\" (UniqueName: \"kubernetes.io/projected/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-kube-api-access-zrwkx\") pod \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " Nov 01 23:01:09.368898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.368758 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-plugins-config-volume\") pod \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\" (UID: \"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e\") " Nov 01 23:01:09.369224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.369191 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-plugins-config-volume" pod "ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" (UID: "ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:09.369743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.369722 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:09.370949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.370918 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-kube-api-access-zrwkx" pod "ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" (UID: "ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e"). InnerVolumeSpecName "kube-api-access-zrwkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:09.458495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.458464 4981 generic.go:356] "Generic (PLEG): container finished" podID="ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" containerID="42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a" exitCode=0 Nov 01 23:01:09.458667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.458512 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" Nov 01 23:01:09.458667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.458547 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" event={"ID":"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e","Type":"ContainerDied","Data":"42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a"} Nov 01 23:01:09.458667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.458642 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5" event={"ID":"ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e","Type":"ContainerDied","Data":"b3faf24466bbfa1139ca86bdba8dc37de41baa6a5185aa0e21a81747b486221a"} Nov 01 23:01:09.458813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.458675 4981 scope.go:122] "RemoveContainer" containerID="42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a" Nov 01 23:01:09.471728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.471686 4981 scope.go:122] "RemoveContainer" containerID="42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a" Nov 01 23:01:09.471978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.471957 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwkx\" (UniqueName: \"kubernetes.io/projected/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e-kube-api-access-zrwkx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:09.472050 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:09.471972 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a\": container with ID starting with 42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a not found: ID does not exist" containerID="42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a" Nov 01 23:01:09.472050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.472001 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a"} err="failed to get container status \"42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a\": rpc error: code = NotFound desc = could not find container \"42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a\": container with ID starting with 42ebb56c8af7f101ae755a62a54bb703610a81be7e5be796b22ad4ce2f07890a not found: ID does not exist" Nov 01 23:01:09.494092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.494061 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5"] Nov 01 23:01:09.497570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.497542 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b48ab8-6dd9545bcc-wlmn5"] Nov 01 23:01:09.654357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:09.654274 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:10.372042 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:10.372012 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:10.736503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:10.736473 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" path="/var/lib/kubelet/pods/ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e/volumes" Nov 01 23:01:10.987974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:10.987892 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:11.100072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:11.100040 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:11.767665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:11.767632 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" podUID="def232dc-e82c-4e0c-851a-bca840f18505" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:01:11.858409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:11.858370 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:12.587182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:12.587143 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:12.777119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:12.777076 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" podUID="fdfcc93a-a688-497a-aabf-801bd24d2159" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:01:13.391896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.391865 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:13.801470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.801438 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 23:01:13.866287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866251 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-data\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.866499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866317 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-envoy\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.866499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866359 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-kube-api-access-86k5t\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-kube-api-access-86k5t\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.866499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866427 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-istio-token\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-istio-token\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.866762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866504 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-certs\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.866762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866536 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42a20f96-6209-41dd-add8-de189e2fba0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a20f96-6209-41dd-add8-de189e2fba0c-istio-podinfo\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.866762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866671 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-data" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:13.866941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.866765 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-socket\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.867250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.867159 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-certs" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:13.867629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.867386 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-credential-socket\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.867629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.867449 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a20f96-6209-41dd-add8-de189e2fba0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a20f96-6209-41dd-add8-de189e2fba0c-istiod-ca-cert\") pod \"42a20f96-6209-41dd-add8-de189e2fba0c\" (UID: \"42a20f96-6209-41dd-add8-de189e2fba0c\") " Nov 01 23:01:13.867629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.867388 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-socket" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:13.868778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.868150 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a20f96-6209-41dd-add8-de189e2fba0c-istiod-ca-cert" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:13.868778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.868684 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42a20f96-6209-41dd-add8-de189e2fba0c-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.868778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.868709 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.868778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.868725 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.868778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.868740 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.869123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.868790 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-credential-socket" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:13.870090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.870064 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-envoy" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:13.870165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.870080 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42a20f96-6209-41dd-add8-de189e2fba0c-istio-podinfo" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:13.870522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.870483 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-kube-api-access-86k5t" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "kube-api-access-86k5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:13.870947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.870923 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-istio-token" pod "42a20f96-6209-41dd-add8-de189e2fba0c" (UID: "42a20f96-6209-41dd-add8-de189e2fba0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:13.970226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.970152 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.970226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.970179 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42a20f96-6209-41dd-add8-de189e2fba0c-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.970226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.970190 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86k5t\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-kube-api-access-86k5t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.970226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.970200 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42a20f96-6209-41dd-add8-de189e2fba0c-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:13.970226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:13.970208 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42a20f96-6209-41dd-add8-de189e2fba0c-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:14.078682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.078648 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:14.524382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.524348 4981 generic.go:356] "Generic (PLEG): container finished" podID="42a20f96-6209-41dd-add8-de189e2fba0c" containerID="0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803" exitCode=0 Nov 01 23:01:14.524627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.524418 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" Nov 01 23:01:14.524627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.524474 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" event={"ID":"42a20f96-6209-41dd-add8-de189e2fba0c","Type":"ContainerDied","Data":"0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803"} Nov 01 23:01:14.524627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.524569 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv" event={"ID":"42a20f96-6209-41dd-add8-de189e2fba0c","Type":"ContainerDied","Data":"b83dc9a14c9cc070bc61d7239bd16f81c0a5529da49f3fca85519fba4fd96c1f"} Nov 01 23:01:14.524811 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.524659 4981 scope.go:122] "RemoveContainer" containerID="0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803" Nov 01 23:01:14.530527 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:14.530498 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:01:14.538011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.537988 4981 scope.go:122] "RemoveContainer" containerID="f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0" Nov 01 23:01:14.548351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.548329 4981 scope.go:122] "RemoveContainer" containerID="0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803" Nov 01 23:01:14.549241 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:14.548662 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803\": container with ID starting with 0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803 not found: ID does not exist" containerID="0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803" Nov 01 23:01:14.549241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.548697 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803"} err="failed to get container status \"0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803\": rpc error: code = NotFound desc = could not find container \"0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803\": container with ID starting with 0973eb17233508bef0a043864d8c192dfe31552b1e8df7ad5cd07b2c6ab65803 not found: ID does not exist" Nov 01 23:01:14.549241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.548722 4981 scope.go:122] "RemoveContainer" containerID="f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0" Nov 01 23:01:14.549461 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:14.549339 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0\": container with ID starting with f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0 not found: ID does not exist" containerID="f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0" Nov 01 23:01:14.549461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.549401 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0"} err="failed to get container status \"f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0\": rpc error: code = NotFound desc = could not find container \"f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0\": container with ID starting with f92bd483f41eaf8f5494e2aa7000ae6d1b8a50eeb335e8253e2283a990bb0ca0 not found: ID does not exist" Nov 01 23:01:14.554671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.554647 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv"] Nov 01 23:01:14.561592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.561559 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-39e952-openshift-default-5979697bf9-9kpwv"] Nov 01 23:01:14.739384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.739351 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" path="/var/lib/kubelet/pods/42a20f96-6209-41dd-add8-de189e2fba0c/volumes" Nov 01 23:01:14.747971 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.747940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:14.978923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:14.978889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:15.368906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:15.368828 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:15.475613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:15.475567 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:16.159767 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:16.159738 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:16.838679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:16.838636 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:17.430141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.430112 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 23:01:17.529948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.529915 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfcc93a-a688-497a-aabf-801bd24d2159-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfcc93a-a688-497a-aabf-801bd24d2159-plugins-config-volume\") pod \"fdfcc93a-a688-497a-aabf-801bd24d2159\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " Nov 01 23:01:17.530130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.530077 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfcc93a-a688-497a-aabf-801bd24d2159-kube-api-access-fb84k\" (UniqueName: \"kubernetes.io/projected/fdfcc93a-a688-497a-aabf-801bd24d2159-kube-api-access-fb84k\") pod \"fdfcc93a-a688-497a-aabf-801bd24d2159\" (UID: \"fdfcc93a-a688-497a-aabf-801bd24d2159\") " Nov 01 23:01:17.530367 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.530345 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfcc93a-a688-497a-aabf-801bd24d2159-plugins-config-volume" pod "fdfcc93a-a688-497a-aabf-801bd24d2159" (UID: "fdfcc93a-a688-497a-aabf-801bd24d2159"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:17.530777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.530757 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfcc93a-a688-497a-aabf-801bd24d2159-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:17.532759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.532734 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfcc93a-a688-497a-aabf-801bd24d2159-kube-api-access-fb84k" pod "fdfcc93a-a688-497a-aabf-801bd24d2159" (UID: "fdfcc93a-a688-497a-aabf-801bd24d2159"). InnerVolumeSpecName "kube-api-access-fb84k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:17.574583 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.574540 4981 generic.go:356] "Generic (PLEG): container finished" podID="fdfcc93a-a688-497a-aabf-801bd24d2159" containerID="bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264" exitCode=0 Nov 01 23:01:17.574739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.574601 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" Nov 01 23:01:17.574739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.574603 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" event={"ID":"fdfcc93a-a688-497a-aabf-801bd24d2159","Type":"ContainerDied","Data":"bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264"} Nov 01 23:01:17.574739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.574658 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g" event={"ID":"fdfcc93a-a688-497a-aabf-801bd24d2159","Type":"ContainerDied","Data":"129e986f8697ba581eefcfd837a0e7726cad88d12b83a33d4cfa263e9b8aa791"} Nov 01 23:01:17.574739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.574683 4981 scope.go:122] "RemoveContainer" containerID="bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264" Nov 01 23:01:17.580330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.580310 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:17.588535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.588513 4981 scope.go:122] "RemoveContainer" containerID="bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264" Nov 01 23:01:17.588843 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:17.588824 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264\": container with ID starting with bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264 not found: ID does not exist" containerID="bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264" Nov 01 23:01:17.588889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.588853 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264"} err="failed to get container status \"bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264\": rpc error: code = NotFound desc = could not find container \"bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264\": container with ID starting with bcf7e7070bf99cdf6f5cef358e0498b55adc48934188bf8479149470db44e264 not found: ID does not exist" Nov 01 23:01:17.613836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.613806 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g"] Nov 01 23:01:17.619774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.619743 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39e952-795944cc55-k8c6g"] Nov 01 23:01:17.632729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:17.632694 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb84k\" (UniqueName: \"kubernetes.io/projected/fdfcc93a-a688-497a-aabf-801bd24d2159-kube-api-access-fb84k\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:18.214388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.214355 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:18.518652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.517949 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:18.695703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.695669 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572"] Nov 01 23:01:18.701630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.701592 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b48ab8-4z572"] Nov 01 23:01:18.732687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.732652 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:18.737901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.737876 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6761e04f-af57-463b-bf6f-c1a554cabe74" path="/var/lib/kubelet/pods/6761e04f-af57-463b-bf6f-c1a554cabe74/volumes" Nov 01 23:01:18.738329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.738314 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfcc93a-a688-497a-aabf-801bd24d2159" path="/var/lib/kubelet/pods/fdfcc93a-a688-497a-aabf-801bd24d2159/volumes" Nov 01 23:01:18.845951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:18.845877 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:19.182340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:19.182311 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:19.533171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:19.533075 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:20.352869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:20.352840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:21.201897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:21.201852 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:21.767249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:21.767217 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" podUID="def232dc-e82c-4e0c-851a-bca840f18505" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:01:22.014994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:22.014961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:22.768936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:22.768902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:23.910718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:23.910665 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:24.900644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:24.900612 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:25.863259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:25.863229 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:26.188440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:26.188408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:26.303464 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:26.303435 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:26.487270 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:26.487189 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t"] Nov 01 23:01:26.487450 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:26.487423 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" containerID="cri-o://e3d95c47c1ec518b4c45ced7758af6c343e81591a070b353533e85651d3d0b58" gracePeriod=30 Nov 01 23:01:26.540824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:26.540788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:26.631996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:26.631964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:27.463516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:27.463476 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:27.662362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:27.662326 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:28.483403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:28.483329 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:29.191659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.191528 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:29.337010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.336983 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 23:01:29.478026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.477995 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def232dc-e82c-4e0c-851a-bca840f18505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def232dc-e82c-4e0c-851a-bca840f18505-plugins-config-volume\") pod \"def232dc-e82c-4e0c-851a-bca840f18505\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " Nov 01 23:01:29.478227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.478135 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def232dc-e82c-4e0c-851a-bca840f18505-kube-api-access-xh6mm\" (UniqueName: \"kubernetes.io/projected/def232dc-e82c-4e0c-851a-bca840f18505-kube-api-access-xh6mm\") pod \"def232dc-e82c-4e0c-851a-bca840f18505\" (UID: \"def232dc-e82c-4e0c-851a-bca840f18505\") " Nov 01 23:01:29.478357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.478335 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def232dc-e82c-4e0c-851a-bca840f18505-plugins-config-volume" pod "def232dc-e82c-4e0c-851a-bca840f18505" (UID: "def232dc-e82c-4e0c-851a-bca840f18505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:29.478834 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.478814 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/def232dc-e82c-4e0c-851a-bca840f18505-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:29.480133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.480115 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def232dc-e82c-4e0c-851a-bca840f18505-kube-api-access-xh6mm" pod "def232dc-e82c-4e0c-851a-bca840f18505" (UID: "def232dc-e82c-4e0c-851a-bca840f18505"). InnerVolumeSpecName "kube-api-access-xh6mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:29.580239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.580156 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6mm\" (UniqueName: \"kubernetes.io/projected/def232dc-e82c-4e0c-851a-bca840f18505-kube-api-access-xh6mm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:29.763541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.763509 4981 generic.go:356] "Generic (PLEG): container finished" podID="def232dc-e82c-4e0c-851a-bca840f18505" containerID="5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1" exitCode=0 Nov 01 23:01:29.763725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.763615 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" Nov 01 23:01:29.763725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.763640 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" event={"ID":"def232dc-e82c-4e0c-851a-bca840f18505","Type":"ContainerDied","Data":"5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1"} Nov 01 23:01:29.763725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.763690 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv" event={"ID":"def232dc-e82c-4e0c-851a-bca840f18505","Type":"ContainerDied","Data":"16a3a16b8e1961f4ce1bc3b3869d6dbdbf5d8ee05b4204f3e6690f279320b572"} Nov 01 23:01:29.763725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.763720 4981 scope.go:122] "RemoveContainer" containerID="5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1" Nov 01 23:01:29.775353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.775339 4981 scope.go:122] "RemoveContainer" containerID="5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1" Nov 01 23:01:29.775615 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:29.775590 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1\": container with ID starting with 5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1 not found: ID does not exist" containerID="5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1" Nov 01 23:01:29.775661 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.775626 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1"} err="failed to get container status \"5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1\": rpc error: code = NotFound desc = could not find container \"5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1\": container with ID starting with 5f1598e8ca7d7be0a844f8f7d0cbb3e3c35c8c0e0d016e4e15c8d2ad327861c1 not found: ID does not exist" Nov 01 23:01:29.789039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.789018 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv"] Nov 01 23:01:29.795188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.795167 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b48ab8-6b5654b5c5-hchfv"] Nov 01 23:01:29.886543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:29.886476 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:30.621959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:30.621918 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:30.737671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:30.737639 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def232dc-e82c-4e0c-851a-bca840f18505" path="/var/lib/kubelet/pods/def232dc-e82c-4e0c-851a-bca840f18505/volumes" Nov 01 23:01:31.344805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.344766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:31.783988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.783961 4981 generic.go:356] "Generic (PLEG): container finished" podID="d827f366-8ae1-425e-98ab-40ed8943f512" containerID="e3d95c47c1ec518b4c45ced7758af6c343e81591a070b353533e85651d3d0b58" exitCode=0 Nov 01 23:01:31.784357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.784029 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" event={"ID":"d827f366-8ae1-425e-98ab-40ed8943f512","Type":"ContainerDied","Data":"e3d95c47c1ec518b4c45ced7758af6c343e81591a070b353533e85651d3d0b58"} Nov 01 23:01:31.784357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.784113 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" event={"ID":"d827f366-8ae1-425e-98ab-40ed8943f512","Type":"ContainerDied","Data":"5a0f948dd239a0c6bafa9faf1772eb39b1917c7bb90119bc9e7ef69fcc36d725"} Nov 01 23:01:31.784357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.784133 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a0f948dd239a0c6bafa9faf1772eb39b1917c7bb90119bc9e7ef69fcc36d725" Nov 01 23:01:31.784357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.784147 4981 scope.go:122] "RemoveContainer" containerID="ceeeba62efcebe1e993adc8be6279e3560a6247ebadf68e4071f33189dd642bf" Nov 01 23:01:31.785246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.785226 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 23:01:31.820230 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820201 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-credential-socket\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820271 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-data\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820288 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-kube-api-access-ndj6x\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-kube-api-access-ndj6x\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820335 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-certs\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820383 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-envoy\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820419 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d827f366-8ae1-425e-98ab-40ed8943f512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d827f366-8ae1-425e-98ab-40ed8943f512-istiod-ca-cert\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820495 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d827f366-8ae1-425e-98ab-40ed8943f512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d827f366-8ae1-425e-98ab-40ed8943f512-istio-podinfo\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820497 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-credential-socket" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:31.820685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820556 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-istio-token\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-istio-token\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.820685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820567 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-data" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:31.820685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820643 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-socket\") pod \"d827f366-8ae1-425e-98ab-40ed8943f512\" (UID: \"d827f366-8ae1-425e-98ab-40ed8943f512\") " Nov 01 23:01:31.821005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820716 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-certs" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:31.821005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.820991 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-socket" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:31.821166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.821089 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d827f366-8ae1-425e-98ab-40ed8943f512-istiod-ca-cert" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:31.822173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.822088 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.822173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.822114 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.822173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.822128 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.822173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.822140 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.822173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.822153 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d827f366-8ae1-425e-98ab-40ed8943f512-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.823164 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.823092 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d827f366-8ae1-425e-98ab-40ed8943f512-istio-podinfo" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:31.824212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.824182 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-istio-token" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:31.824751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.824717 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-kube-api-access-ndj6x" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "kube-api-access-ndj6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:31.825253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.825233 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-envoy" pod "d827f366-8ae1-425e-98ab-40ed8943f512" (UID: "d827f366-8ae1-425e-98ab-40ed8943f512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:31.923800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.923774 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndj6x\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-kube-api-access-ndj6x\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.923947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.923801 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d827f366-8ae1-425e-98ab-40ed8943f512-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.923947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.923811 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d827f366-8ae1-425e-98ab-40ed8943f512-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:31.923947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:31.923820 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d827f366-8ae1-425e-98ab-40ed8943f512-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:32.208774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:32.208686 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:32.793453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:32.793419 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t" Nov 01 23:01:32.835861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:32.835827 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t"] Nov 01 23:01:32.839905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:32.839873 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b48ab8-openshift-default-6ff79c5d59-6q47t"] Nov 01 23:01:33.079230 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:33.079146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:33.750806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:33.750775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:34.451188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:34.451152 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:34.749641 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:34.749532 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" path="/var/lib/kubelet/pods/d827f366-8ae1-425e-98ab-40ed8943f512/volumes" Nov 01 23:01:35.318636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:35.318602 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:36.038670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:36.038634 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:36.857170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:36.857137 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:37.608198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:37.608166 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:38.385991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:38.385959 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:39.166322 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:39.166286 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:40.098364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:40.098334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:40.464868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:40.464836 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr"] Nov 01 23:01:40.465332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:40.465080 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" containerID="cri-o://ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22" gracePeriod=30 Nov 01 23:01:40.885405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:40.885324 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:41.441349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:41.441314 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh"] Nov 01 23:01:41.441536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:41.441511 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" containerID="cri-o://253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5" gracePeriod=30 Nov 01 23:01:41.580798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:41.580763 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq"] Nov 01 23:01:41.581287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:41.581045 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" containerID="cri-o://b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21" gracePeriod=30 Nov 01 23:01:41.859160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:41.859079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:42.499485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.499452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:42.666810 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.666778 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g"] Nov 01 23:01:42.667958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.667938 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" Nov 01 23:01:42.668011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.667962 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" Nov 01 23:01:42.668011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.667981 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" containerName="payload-processing" Nov 01 23:01:42.668011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.667990 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" containerName="payload-processing" Nov 01 23:01:42.668011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668004 4981 cpu_manager.go:414] "containerMap: removing container" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668011 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668026 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fdfcc93a-a688-497a-aabf-801bd24d2159" containerName="payload-pre-processing" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668033 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfcc93a-a688-497a-aabf-801bd24d2159" containerName="payload-pre-processing" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668058 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6761e04f-af57-463b-bf6f-c1a554cabe74" containerName="revoke-keys" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668064 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6761e04f-af57-463b-bf6f-c1a554cabe74" containerName="revoke-keys" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668081 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" containerName="payload-processing" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668088 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" containerName="payload-processing" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668111 4981 cpu_manager.go:414] "containerMap: removing container" podUID="def232dc-e82c-4e0c-851a-bca840f18505" containerName="payload-pre-processing" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668119 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def232dc-e82c-4e0c-851a-bca840f18505" containerName="payload-pre-processing" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668133 4981 cpu_manager.go:414] "containerMap: removing container" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" Nov 01 23:01:42.668148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668140 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668155 4981 cpu_manager.go:414] "containerMap: removing container" podUID="369ead52-8316-4afd-9b8f-ab992cd76b7d" containerName="payload-pre-processing" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668163 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="369ead52-8316-4afd-9b8f-ab992cd76b7d" containerName="payload-pre-processing" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668185 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668192 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668358 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668365 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e21a98-7a1e-45d7-a97f-8344eefe7843" containerName="payload-processing" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668371 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668377 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfcc93a-a688-497a-aabf-801bd24d2159" containerName="payload-pre-processing" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668385 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6761e04f-af57-463b-bf6f-c1a554cabe74" containerName="revoke-keys" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668393 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffba04c8-57c1-4023-9e7d-5b7cc8e6c60e" containerName="payload-processing" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668401 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="def232dc-e82c-4e0c-851a-bca840f18505" containerName="payload-pre-processing" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668407 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a20f96-6209-41dd-add8-de189e2fba0c" containerName="istio-proxy" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668413 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="369ead52-8316-4afd-9b8f-ab992cd76b7d" containerName="payload-pre-processing" Nov 01 23:01:42.668485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.668420 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d827f366-8ae1-425e-98ab-40ed8943f512" containerName="istio-proxy" Nov 01 23:01:42.674353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.674329 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.678107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.678083 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g"] Nov 01 23:01:42.686406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.686383 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a9f94e\"/\"rate-test-model-a9f94e-kserve-self-signed-certs\"" Nov 01 23:01:42.686538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.686422 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a9f94e\"/\"openshift-service-ca.crt\"" Nov 01 23:01:42.686538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.686500 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a9f94e\"/\"kube-root-ca.crt\"" Nov 01 23:01:42.686733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.686397 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a9f94e\"/\"default-dockercfg-cz68m\"" Nov 01 23:01:42.703870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.703840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:42.825357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.825254 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-dshm\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.825980 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.825938 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tmp-dir\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.826193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.826170 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-model-cache\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.826332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.826316 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tls-certs\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.826557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.826538 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j5v\" (UniqueName: \"kubernetes.io/projected/a5f8dbc0-0979-4803-9322-6e9ac529e96e-kube-api-access-k6j5v\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.826740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.826717 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-home\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.928090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.928054 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-model-cache\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.928315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.928127 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tls-certs\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.928315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.928257 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j5v\" (UniqueName: \"kubernetes.io/projected/a5f8dbc0-0979-4803-9322-6e9ac529e96e-kube-api-access-k6j5v\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.928407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.928313 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-home\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.928407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.928397 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-dshm\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.928488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.928464 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tmp-dir\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.928545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.928521 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-model-cache\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.930002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.929630 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-home\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.930183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.930163 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tmp-dir\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.932332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.932306 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tls-certs\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.932448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.932423 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-dshm\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.938833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.938804 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j5v\" (UniqueName: \"kubernetes.io/projected/a5f8dbc0-0979-4803-9322-6e9ac529e96e-kube-api-access-k6j5v\") pod \"rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:42.990358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:42.990326 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:43.275393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:43.275363 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g"] Nov 01 23:01:43.275848 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:01:43.275823 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f8dbc0_0979_4803_9322_6e9ac529e96e.slice/crio-201aafc432eccaa73c6afb5946f788f74c7d3cf30c87058d7053d67009ce4169 WatchSource:0}: Error finding container 201aafc432eccaa73c6afb5946f788f74c7d3cf30c87058d7053d67009ce4169: Status 404 returned error can't find the container with id 201aafc432eccaa73c6afb5946f788f74c7d3cf30c87058d7053d67009ce4169 Nov 01 23:01:43.380107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:43.380076 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:43.925112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:43.925066 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" event={"ID":"a5f8dbc0-0979-4803-9322-6e9ac529e96e","Type":"ContainerStarted","Data":"9e1294fe7f527ab606818d617e2b62b9dad59a6def5c331bbdbcb823a65ad030"} Nov 01 23:01:43.925112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:43.925110 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" event={"ID":"a5f8dbc0-0979-4803-9322-6e9ac529e96e","Type":"ContainerStarted","Data":"201aafc432eccaa73c6afb5946f788f74c7d3cf30c87058d7053d67009ce4169"} Nov 01 23:01:43.950184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:43.950134 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" podStartSLOduration=1.950118252 podStartE2EDuration="1.950118252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:43.948625597 +0000 UTC m=+2269.376503527" watchObservedRunningTime="2026-11-01 23:01:43.950118252 +0000 UTC m=+2269.377996181" Nov 01 23:01:43.990509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:43.990481 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:44.067115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:44.067084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:44.526504 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:44.526469 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:01:44.699269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:44.699232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:45.341130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.341094 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:45.747684 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.747659 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 23:01:45.755622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.755564 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:01:45.755746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.755643 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:01:45.786120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786088 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-certs\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.786300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786209 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-envoy\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786385 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-credential-socket\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786419 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-socket\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786507 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-data\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786535 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-kube-api-access-6g2bw\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-kube-api-access-6g2bw\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786568 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4837ce-9330-4371-8e95-1dccb1851627-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4837ce-9330-4371-8e95-1dccb1851627-istiod-ca-cert\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786700 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f4837ce-9330-4371-8e95-1dccb1851627-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4837ce-9330-4371-8e95-1dccb1851627-istio-podinfo\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786735 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-istio-token\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-istio-token\") pod \"6f4837ce-9330-4371-8e95-1dccb1851627\" (UID: \"6f4837ce-9330-4371-8e95-1dccb1851627\") " Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.786944 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-data" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.787627 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.788149 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-credential-socket" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.788305 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-socket" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:45.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.788871 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4837ce-9330-4371-8e95-1dccb1851627-istiod-ca-cert" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:45.790244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.790217 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-certs" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:45.790388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.790368 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-istio-token" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:45.791206 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.791180 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-kube-api-access-6g2bw" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "kube-api-access-6g2bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:45.792594 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.792557 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f4837ce-9330-4371-8e95-1dccb1851627-istio-podinfo" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:45.792715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.792702 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-envoy" pod "6f4837ce-9330-4371-8e95-1dccb1851627" (UID: "6f4837ce-9330-4371-8e95-1dccb1851627"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:45.889120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889034 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4837ce-9330-4371-8e95-1dccb1851627-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.889120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889065 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.889120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889079 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.889120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889092 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.889120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889104 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.889565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889160 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4837ce-9330-4371-8e95-1dccb1851627-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.889565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889174 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g2bw\" (UniqueName: \"kubernetes.io/projected/6f4837ce-9330-4371-8e95-1dccb1851627-kube-api-access-6g2bw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.889565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.889196 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4837ce-9330-4371-8e95-1dccb1851627-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:45.943256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.943227 4981 generic.go:356] "Generic (PLEG): container finished" podID="6f4837ce-9330-4371-8e95-1dccb1851627" containerID="ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22" exitCode=0 Nov 01 23:01:45.943414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.943274 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" Nov 01 23:01:45.943414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.943311 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" event={"ID":"6f4837ce-9330-4371-8e95-1dccb1851627","Type":"ContainerDied","Data":"ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22"} Nov 01 23:01:45.943414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.943366 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr" event={"ID":"6f4837ce-9330-4371-8e95-1dccb1851627","Type":"ContainerDied","Data":"b44168364c6f795ab1df77de582b32ffa91982621c8da4491dc854c79d8874af"} Nov 01 23:01:45.943414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.943392 4981 scope.go:122] "RemoveContainer" containerID="ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22" Nov 01 23:01:45.955533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.955509 4981 scope.go:122] "RemoveContainer" containerID="ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22" Nov 01 23:01:45.955859 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:45.955827 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22\": container with ID starting with ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22 not found: ID does not exist" containerID="ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22" Nov 01 23:01:45.955952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.955854 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22"} err="failed to get container status \"ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22\": rpc error: code = NotFound desc = could not find container \"ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22\": container with ID starting with ab8b946c4155a79724448d02f4efae2ba4add985ec6bc0b9d00bcb0255e43c22 not found: ID does not exist" Nov 01 23:01:45.970849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.970823 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr"] Nov 01 23:01:45.975338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:45.975307 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e26733-openshift-default-7dbf776bff-wkrvr"] Nov 01 23:01:46.068897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.068863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:46.726634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.726523 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 23:01:46.738325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.738296 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" path="/var/lib/kubelet/pods/6f4837ce-9330-4371-8e95-1dccb1851627/volumes" Nov 01 23:01:46.790760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.790730 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:46.804697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.804654 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-data\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.804868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.804738 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-kube-api-access-hrnpx\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-kube-api-access-hrnpx\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.804868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.804837 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-certs\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.805002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.804936 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-credential-socket\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.805002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.804971 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-podinfo\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.805121 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.805029 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-envoy\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.805121 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.805057 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-token\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-token\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.805238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.805140 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7731aca5-5577-4ec4-ba55-6c379ab1be21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7731aca5-5577-4ec4-ba55-6c379ab1be21-istiod-ca-cert\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.805304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.805260 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-socket\") pod \"7731aca5-5577-4ec4-ba55-6c379ab1be21\" (UID: \"7731aca5-5577-4ec4-ba55-6c379ab1be21\") " Nov 01 23:01:46.805715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.805679 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-credential-socket" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.806080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.806060 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-data" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.806213 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.806111 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.807013 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.806838 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-certs" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.807439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.807174 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-socket" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.809023 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.808971 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7731aca5-5577-4ec4-ba55-6c379ab1be21-istiod-ca-cert" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:46.809133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.809088 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-podinfo" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:46.815040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.815016 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-envoy" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.817033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.817006 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-kube-api-access-hrnpx" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "kube-api-access-hrnpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.818726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.818704 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-token" pod "7731aca5-5577-4ec4-ba55-6c379ab1be21" (UID: "7731aca5-5577-4ec4-ba55-6c379ab1be21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.860894 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.860860 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907071 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-certs\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907120 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-kube-api-access-5vhhq\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-kube-api-access-5vhhq\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907177 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-podinfo\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907226 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-data\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907380 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istiod-ca-cert\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907407 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-socket\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907455 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-envoy\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907513 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-credential-socket\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.907681 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-token\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-token\") pod \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\" (UID: \"dc735e8a-9d22-4cbc-868b-be4de72ccbe9\") " Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908462 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908483 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908498 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908513 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908528 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7731aca5-5577-4ec4-ba55-6c379ab1be21-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908544 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908557 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7731aca5-5577-4ec4-ba55-6c379ab1be21-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.908584 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnpx\" (UniqueName: \"kubernetes.io/projected/7731aca5-5577-4ec4-ba55-6c379ab1be21-kube-api-access-hrnpx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.909422 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istiod-ca-cert" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:01:46.913597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.910182 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-certs" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.921062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.915459 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-socket" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.921062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.916234 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-data" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.921062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.920156 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-credential-socket" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.921338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.921239 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-kube-api-access-5vhhq" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "kube-api-access-5vhhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.923438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.923400 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-podinfo" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:01:46.923833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.923806 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-token" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:01:46.926040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.926012 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-envoy" pod "dc735e8a-9d22-4cbc-868b-be4de72ccbe9" (UID: "dc735e8a-9d22-4cbc-868b-be4de72ccbe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:01:46.958798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.958756 4981 generic.go:356] "Generic (PLEG): container finished" podID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerID="253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5" exitCode=0 Nov 01 23:01:46.958963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.958911 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" event={"ID":"7731aca5-5577-4ec4-ba55-6c379ab1be21","Type":"ContainerDied","Data":"253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5"} Nov 01 23:01:46.959049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.958966 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" event={"ID":"7731aca5-5577-4ec4-ba55-6c379ab1be21","Type":"ContainerDied","Data":"44265ad8a37e3a015c6c203d0bacef4b862181c84f7454c6212406843ae49a1d"} Nov 01 23:01:46.959049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.958998 4981 scope.go:122] "RemoveContainer" containerID="253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5" Nov 01 23:01:46.959162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.959125 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh" Nov 01 23:01:46.976243 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.976209 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerID="b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21" exitCode=0 Nov 01 23:01:46.976554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.976515 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" event={"ID":"dc735e8a-9d22-4cbc-868b-be4de72ccbe9","Type":"ContainerDied","Data":"b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21"} Nov 01 23:01:46.976744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.976639 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" event={"ID":"dc735e8a-9d22-4cbc-868b-be4de72ccbe9","Type":"ContainerDied","Data":"3e6dccc6bcf992f67fe6a3a9c4a7e1d655a98cd4ddedbede99de22bf9ec658af"} Nov 01 23:01:46.976952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:46.976932 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012299 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012330 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012339 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012348 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012358 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012367 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012375 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012384 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012392 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vhhq\" (UniqueName: \"kubernetes.io/projected/dc735e8a-9d22-4cbc-868b-be4de72ccbe9-kube-api-access-5vhhq\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:01:47.012480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.012415 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh"] Nov 01 23:01:47.018849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.018819 4981 scope.go:122] "RemoveContainer" containerID="253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5" Nov 01 23:01:47.020443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.018828 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-jpqjh"] Nov 01 23:01:47.020911 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:47.020833 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5\": container with ID starting with 253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5 not found: ID does not exist" containerID="253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5" Nov 01 23:01:47.021053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.021032 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5"} err="failed to get container status \"253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5\": rpc error: code = NotFound desc = could not find container \"253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5\": container with ID starting with 253a7f88f9f659c81413600df5c7cdd80399720b99cd4a9965ebc053892a32b5 not found: ID does not exist" Nov 01 23:01:47.021131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.021121 4981 scope.go:122] "RemoveContainer" containerID="b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21" Nov 01 23:01:47.033498 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.033460 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq"] Nov 01 23:01:47.038009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.037982 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-4g7rq"] Nov 01 23:01:47.039926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.039906 4981 scope.go:122] "RemoveContainer" containerID="b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21" Nov 01 23:01:47.040323 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:01:47.040302 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21\": container with ID starting with b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21 not found: ID does not exist" containerID="b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21" Nov 01 23:01:47.040429 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.040331 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21"} err="failed to get container status \"b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21\": rpc error: code = NotFound desc = could not find container \"b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21\": container with ID starting with b38d8b543817b76f218cc0f833a12ffeb43a3b0088b42d2543f740f2b17b2a21 not found: ID does not exist" Nov 01 23:01:47.651893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:47.651860 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:48.458652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:48.458617 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:48.736876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:48.736785 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" path="/var/lib/kubelet/pods/7731aca5-5577-4ec4-ba55-6c379ab1be21/volumes" Nov 01 23:01:48.737318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:48.737303 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" path="/var/lib/kubelet/pods/dc735e8a-9d22-4cbc-868b-be4de72ccbe9/volumes" Nov 01 23:01:49.160715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:49.160634 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:49.930560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:49.930528 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:50.802048 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:50.802015 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:51.468278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:51.468227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:52.069217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:52.069160 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:52.697274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:52.697232 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:52.991476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:52.991385 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:53.004398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:53.004367 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:53.057542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:53.057508 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:01:53.382054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:53.381975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:54.066991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:54.066963 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:54.673097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:54.673067 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:55.349496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:55.349459 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:56.047387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:56.047352 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:56.841613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:56.841560 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:57.757676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:57.757640 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:58.760882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:58.760844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:01:59.658570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:01:59.658527 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:00.369932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:00.369896 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:01.082409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.082362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:01.622176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.622141 4981 scope.go:122] "RemoveContainer" containerID="74c344857776e8e21494f798a0fe2222b232599e5387a2bb0a2f9c7d9e1db4a5" Nov 01 23:02:01.632986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.632963 4981 scope.go:122] "RemoveContainer" containerID="cb9ebd4908b83e6d790d1a03deb99ffded0f6ab514b8d99812fa4358e507deb0" Nov 01 23:02:01.643809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.643782 4981 scope.go:122] "RemoveContainer" containerID="fe9ba8e746e5d7f7dd12aa2c8f81040f931b651dc43ebb5382b88a1580abbe4d" Nov 01 23:02:01.654524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.654500 4981 scope.go:122] "RemoveContainer" containerID="e3d95c47c1ec518b4c45ced7758af6c343e81591a070b353533e85651d3d0b58" Nov 01 23:02:01.665752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.665728 4981 scope.go:122] "RemoveContainer" containerID="459390a1f57a294494e4ad60d45f7db1241c2e554721b68bb8ad1c29dd69df5a" Nov 01 23:02:01.676173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.676148 4981 scope.go:122] "RemoveContainer" containerID="002b42c35ab96d4ea3eaacc8b787af0c3f3c65acc50b4a01d33e83f484250703" Nov 01 23:02:01.685905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.685884 4981 scope.go:122] "RemoveContainer" containerID="66e53ef525cad2e30c6b34082dc7fc062ccb9dad3c657c5d9272ce93f4a8398a" Nov 01 23:02:01.797455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:01.797426 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:02.253346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.253313 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r"] Nov 01 23:02:02.254885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.254850 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" Nov 01 23:02:02.254885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.254880 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" Nov 01 23:02:02.255098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.254922 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" Nov 01 23:02:02.255098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.254930 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" Nov 01 23:02:02.255098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.254976 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" Nov 01 23:02:02.255098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.254984 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" Nov 01 23:02:02.255316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.255174 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4837ce-9330-4371-8e95-1dccb1851627" containerName="istio-proxy" Nov 01 23:02:02.255316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.255196 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7731aca5-5577-4ec4-ba55-6c379ab1be21" containerName="istio-proxy" Nov 01 23:02:02.255316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.255211 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc735e8a-9d22-4cbc-868b-be4de72ccbe9" containerName="istio-proxy" Nov 01 23:02:02.268207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.268176 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.270958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.270932 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r"] Nov 01 23:02:02.274381 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.274352 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:02.274570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.274550 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:02:02.369155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.369114 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4vf\" (UniqueName: \"kubernetes.io/projected/db084668-3f72-4b0d-92b1-49e1b732e361-kube-api-access-tw4vf\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.369344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.369222 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.369419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.369406 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.470986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.470950 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.471178 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.471120 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.471227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.471203 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4vf\" (UniqueName: \"kubernetes.io/projected/db084668-3f72-4b0d-92b1-49e1b732e361-kube-api-access-tw4vf\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.471889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.471868 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.473475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.473453 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.479885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.479863 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4vf\" (UniqueName: \"kubernetes.io/projected/db084668-3f72-4b0d-92b1-49e1b732e361-kube-api-access-tw4vf\") pod \"maas-api-revoke-sub-3fd53567-qw92r\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.588974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.588895 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:02.595617 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.595590 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:02.888439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:02.888398 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r"] Nov 01 23:02:02.888915 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:02:02.888495 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb084668_3f72_4b0d_92b1_49e1b732e361.slice/crio-e9f47e6d081eadadd6ab2d4a97f12dd73b9b5a15dfa8e4525f9e4a4746c779fa WatchSource:0}: Error finding container e9f47e6d081eadadd6ab2d4a97f12dd73b9b5a15dfa8e4525f9e4a4746c779fa: Status 404 returned error can't find the container with id e9f47e6d081eadadd6ab2d4a97f12dd73b9b5a15dfa8e4525f9e4a4746c779fa Nov 01 23:02:03.147378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:03.147291 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" event={"ID":"db084668-3f72-4b0d-92b1-49e1b732e361","Type":"ContainerStarted","Data":"e9f47e6d081eadadd6ab2d4a97f12dd73b9b5a15dfa8e4525f9e4a4746c779fa"} Nov 01 23:02:03.286976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:03.286950 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:03.976017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:03.975972 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:04.157386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:04.157291 4981 generic.go:356] "Generic (PLEG): container finished" podID="db084668-3f72-4b0d-92b1-49e1b732e361" containerID="1d9c5a820cecdb7adfaf077d8bc3640c992ab3ff67c931c268403cd990e1114c" exitCode=0 Nov 01 23:02:04.157386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:04.157337 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" event={"ID":"db084668-3f72-4b0d-92b1-49e1b732e361","Type":"ContainerDied","Data":"1d9c5a820cecdb7adfaf077d8bc3640c992ab3ff67c931c268403cd990e1114c"} Nov 01 23:02:04.757379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:04.757348 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:05.328921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.328897 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:05.422144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.422062 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db084668-3f72-4b0d-92b1-49e1b732e361-kube-api-access-tw4vf\" (UniqueName: \"kubernetes.io/projected/db084668-3f72-4b0d-92b1-49e1b732e361-kube-api-access-tw4vf\") pod \"db084668-3f72-4b0d-92b1-49e1b732e361\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " Nov 01 23:02:05.422291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.422186 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-openshift-ca\") pod \"db084668-3f72-4b0d-92b1-49e1b732e361\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " Nov 01 23:02:05.422291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.422238 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-xks-ca\") pod \"db084668-3f72-4b0d-92b1-49e1b732e361\" (UID: \"db084668-3f72-4b0d-92b1-49e1b732e361\") " Nov 01 23:02:05.422633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.422568 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-openshift-ca" pod "db084668-3f72-4b0d-92b1-49e1b732e361" (UID: "db084668-3f72-4b0d-92b1-49e1b732e361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:05.422971 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.422953 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:05.424628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.424599 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-xks-ca" pod "db084668-3f72-4b0d-92b1-49e1b732e361" (UID: "db084668-3f72-4b0d-92b1-49e1b732e361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:05.424722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.424626 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db084668-3f72-4b0d-92b1-49e1b732e361-kube-api-access-tw4vf" pod "db084668-3f72-4b0d-92b1-49e1b732e361" (UID: "db084668-3f72-4b0d-92b1-49e1b732e361"). InnerVolumeSpecName "kube-api-access-tw4vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:05.473758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.473726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:05.542363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.542313 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4vf\" (UniqueName: \"kubernetes.io/projected/db084668-3f72-4b0d-92b1-49e1b732e361-kube-api-access-tw4vf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:05.542363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.542354 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db084668-3f72-4b0d-92b1-49e1b732e361-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:05.946253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.946215 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb"] Nov 01 23:02:05.947501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.947478 4981 cpu_manager.go:414] "containerMap: removing container" podUID="db084668-3f72-4b0d-92b1-49e1b732e361" containerName="revoke-keys" Nov 01 23:02:05.947570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.947505 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db084668-3f72-4b0d-92b1-49e1b732e361" containerName="revoke-keys" Nov 01 23:02:05.947763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.947746 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="db084668-3f72-4b0d-92b1-49e1b732e361" containerName="revoke-keys" Nov 01 23:02:05.951780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.951760 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:05.963151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.963096 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-56b833\"/\"default-dockercfg-htfwv\"" Nov 01 23:02:05.963419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.963394 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-56b833\"/\"rate-test-model-56b833-kserve-self-signed-certs\"" Nov 01 23:02:05.963534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.963489 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-56b833\"/\"kube-root-ca.crt\"" Nov 01 23:02:05.964599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.963552 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-56b833\"/\"openshift-service-ca.crt\"" Nov 01 23:02:05.964599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.963682 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb"] Nov 01 23:02:05.985260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:05.985227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:06.050165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.050126 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-home\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.050377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.050204 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tk9s\" (UniqueName: \"kubernetes.io/projected/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-kube-api-access-2tk9s\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.050377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.050337 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tls-certs\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.050510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.050468 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tmp-dir\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.050611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.050519 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-model-cache\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.050724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.050692 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-dshm\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.152646 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-dshm\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.152794 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-home\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.152838 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tk9s\" (UniqueName: \"kubernetes.io/projected/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-kube-api-access-2tk9s\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.152916 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tls-certs\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.153020 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tmp-dir\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.153058 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-model-cache\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.153634 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-model-cache\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.154942 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-home\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.188670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.157115 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tmp-dir\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.189502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.189472 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-dshm\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.192457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.190729 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tls-certs\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.193227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.193204 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:06.201345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.199550 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" event={"ID":"db084668-3f72-4b0d-92b1-49e1b732e361","Type":"ContainerDied","Data":"e9f47e6d081eadadd6ab2d4a97f12dd73b9b5a15dfa8e4525f9e4a4746c779fa"} Nov 01 23:02:06.201345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.199673 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9f47e6d081eadadd6ab2d4a97f12dd73b9b5a15dfa8e4525f9e4a4746c779fa" Nov 01 23:02:06.201345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.199764 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r" Nov 01 23:02:06.213695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.213664 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tk9s\" (UniqueName: \"kubernetes.io/projected/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-kube-api-access-2tk9s\") pod \"rate-test-model-56b833-kserve-64776bd665-5g2kb\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.274523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.274234 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:06.448262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.448227 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7"] Nov 01 23:02:06.448804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.448518 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" containerID="cri-o://1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3" gracePeriod=30 Nov 01 23:02:06.569750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:06.566437 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb"] Nov 01 23:02:07.094983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:07.094948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:07.212857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:07.212818 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" event={"ID":"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9","Type":"ContainerStarted","Data":"5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9"} Nov 01 23:02:07.213053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:07.212870 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" event={"ID":"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9","Type":"ContainerStarted","Data":"b93ea676dc601758b920acdb4dace66bc68888e33efbed272bbb6813002ddeca"} Nov 01 23:02:07.231689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:07.231635 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" podStartSLOduration=2.23161888 podStartE2EDuration="2.23161888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:07.230139173 +0000 UTC m=+2292.658017106" watchObservedRunningTime="2026-11-01 23:02:07.23161888 +0000 UTC m=+2292.659496815" Nov 01 23:02:07.274740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:07.274692 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:07.957771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:07.957736 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:08.583569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:08.583533 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:09.460731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:09.460699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:10.058847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:10.058818 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:10.781708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:10.781669 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:11.494017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.493989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:11.744704 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.744677 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 23:02:11.865569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.865533 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-kube-api-access-t84tc\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-kube-api-access-t84tc\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.865635 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8220b2c2-c473-4e28-9caf-e20b3fd61537-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8220b2c2-c473-4e28-9caf-e20b3fd61537-istiod-ca-cert\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.865741 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-podinfo\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.865799 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-socket\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.865971 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-certs\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866027 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-envoy\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866084 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-credential-socket\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866129 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-token\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-token\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866232 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-data\") pod \"8220b2c2-c473-4e28-9caf-e20b3fd61537\" (UID: \"8220b2c2-c473-4e28-9caf-e20b3fd61537\") " Nov 01 23:02:11.866426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866122 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-socket" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:11.866426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866152 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8220b2c2-c473-4e28-9caf-e20b3fd61537-istiod-ca-cert" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:11.866426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866326 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-credential-socket" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:11.866709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866552 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-certs" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:11.866709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866626 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-data" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:11.867017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.866996 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.867062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.867026 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.867062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.867041 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8220b2c2-c473-4e28-9caf-e20b3fd61537-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.867062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.867054 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.867155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.867068 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.868236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.868211 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-podinfo" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:02:11.868385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.868233 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-kube-api-access-t84tc" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "kube-api-access-t84tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:11.868558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.868531 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-token" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:11.868558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.868557 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-envoy" pod "8220b2c2-c473-4e28-9caf-e20b3fd61537" (UID: "8220b2c2-c473-4e28-9caf-e20b3fd61537"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:02:11.968232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.968201 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.968232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.968230 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.968456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.968240 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t84tc\" (UniqueName: \"kubernetes.io/projected/8220b2c2-c473-4e28-9caf-e20b3fd61537-kube-api-access-t84tc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:11.968456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:11.968250 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8220b2c2-c473-4e28-9caf-e20b3fd61537-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:12.260705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.260673 4981 generic.go:356] "Generic (PLEG): container finished" podID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerID="1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3" exitCode=0 Nov 01 23:02:12.260863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.260724 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" Nov 01 23:02:12.260863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.260754 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" event={"ID":"8220b2c2-c473-4e28-9caf-e20b3fd61537","Type":"ContainerDied","Data":"1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3"} Nov 01 23:02:12.260863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.260809 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7" event={"ID":"8220b2c2-c473-4e28-9caf-e20b3fd61537","Type":"ContainerDied","Data":"f6c00e66468bd7d48219ebd990381de2f2a41e2d14e9ad5d92907c2c6b258a76"} Nov 01 23:02:12.260863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.260836 4981 scope.go:122] "RemoveContainer" containerID="1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3" Nov 01 23:02:12.266639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.266616 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:12.274218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.274200 4981 scope.go:122] "RemoveContainer" containerID="80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587" Nov 01 23:02:12.284736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.284716 4981 scope.go:122] "RemoveContainer" containerID="1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3" Nov 01 23:02:12.285021 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:02:12.285002 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3\": container with ID starting with 1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3 not found: ID does not exist" containerID="1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3" Nov 01 23:02:12.285092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.285027 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3"} err="failed to get container status \"1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3\": rpc error: code = NotFound desc = could not find container \"1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3\": container with ID starting with 1650ef1c7e9500ef09f369755608a6a92d4ef55a0fd522d59520270bc5e79ca3 not found: ID does not exist" Nov 01 23:02:12.285092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.285043 4981 scope.go:122] "RemoveContainer" containerID="80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587" Nov 01 23:02:12.285295 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:02:12.285277 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587\": container with ID starting with 80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587 not found: ID does not exist" containerID="80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587" Nov 01 23:02:12.285336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.285301 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587"} err="failed to get container status \"80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587\": rpc error: code = NotFound desc = could not find container \"80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587\": container with ID starting with 80142082f64c649b172785ac99bd0df7f4c6ea4dba6aa64c95b96986a020a587 not found: ID does not exist" Nov 01 23:02:12.293258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.293238 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7"] Nov 01 23:02:12.299947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.299927 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-64b196-openshift-default-8d977c5d5-g44r7"] Nov 01 23:02:12.443319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.443282 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4"] Nov 01 23:02:12.444129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.444106 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" Nov 01 23:02:12.444129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.444128 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" Nov 01 23:02:12.444318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.444172 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" Nov 01 23:02:12.444318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.444181 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" Nov 01 23:02:12.444435 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.444364 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" Nov 01 23:02:12.444435 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.444385 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" containerName="istio-proxy" Nov 01 23:02:12.450412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.449515 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.453342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.453315 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:02:12.453467 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.453387 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:12.456494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.456472 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4"] Nov 01 23:02:12.577592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.577481 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjjwv\" (UniqueName: \"kubernetes.io/projected/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-kube-api-access-hjjwv\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.577592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.577517 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.577592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.577558 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.679240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.679204 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.679484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.679466 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjjwv\" (UniqueName: \"kubernetes.io/projected/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-kube-api-access-hjjwv\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.679613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.679492 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.680146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.680126 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.681614 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.681593 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.687286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.687263 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjjwv\" (UniqueName: \"kubernetes.io/projected/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-kube-api-access-hjjwv\") pod \"maas-api-revoke-sub-03f0889c-n6gg4\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.738409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.738368 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8220b2c2-c473-4e28-9caf-e20b3fd61537" path="/var/lib/kubelet/pods/8220b2c2-c473-4e28-9caf-e20b3fd61537/volumes" Nov 01 23:02:12.770005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.769972 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:12.976008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:12.975979 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:13.058239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:13.058204 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4"] Nov 01 23:02:13.063325 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:02:13.063289 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8af5ae3_5412_49ee_b1c6_c790ce6b655d.slice/crio-f8b167f950e5176c630e1b98e854f75e0585923843033b9f92965f8b14d13f0a WatchSource:0}: Error finding container f8b167f950e5176c630e1b98e854f75e0585923843033b9f92965f8b14d13f0a: Status 404 returned error can't find the container with id f8b167f950e5176c630e1b98e854f75e0585923843033b9f92965f8b14d13f0a Nov 01 23:02:13.270402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:13.270368 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" event={"ID":"b8af5ae3-5412-49ee-b1c6-c790ce6b655d","Type":"ContainerStarted","Data":"f8b167f950e5176c630e1b98e854f75e0585923843033b9f92965f8b14d13f0a"} Nov 01 23:02:13.773771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:13.773727 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:14.280174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:14.280140 4981 generic.go:356] "Generic (PLEG): container finished" podID="b8af5ae3-5412-49ee-b1c6-c790ce6b655d" containerID="d04fe4332f4d22b72c4b9c7e9af4f6369b1d88df94a03f7272b647ff17e376a8" exitCode=0 Nov 01 23:02:14.280605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:14.280208 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" event={"ID":"b8af5ae3-5412-49ee-b1c6-c790ce6b655d","Type":"ContainerDied","Data":"d04fe4332f4d22b72c4b9c7e9af4f6369b1d88df94a03f7272b647ff17e376a8"} Nov 01 23:02:14.480473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:14.480430 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:14.554053 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:02:14.554017 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:02:14.883151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:14.883048 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:15.266967 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.266937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:15.489501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.489474 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:15.529043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.528971 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-kube-api-access-hjjwv\" (UniqueName: \"kubernetes.io/projected/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-kube-api-access-hjjwv\") pod \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " Nov 01 23:02:15.529198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.529115 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-openshift-ca\") pod \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " Nov 01 23:02:15.529198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.529141 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-xks-ca\") pod \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\" (UID: \"b8af5ae3-5412-49ee-b1c6-c790ce6b655d\") " Nov 01 23:02:15.529548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.529517 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-openshift-ca" pod "b8af5ae3-5412-49ee-b1c6-c790ce6b655d" (UID: "b8af5ae3-5412-49ee-b1c6-c790ce6b655d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:15.529969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.529942 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:15.531282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.531251 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-kube-api-access-hjjwv" pod "b8af5ae3-5412-49ee-b1c6-c790ce6b655d" (UID: "b8af5ae3-5412-49ee-b1c6-c790ce6b655d"). InnerVolumeSpecName "kube-api-access-hjjwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:15.531370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.531333 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-xks-ca" pod "b8af5ae3-5412-49ee-b1c6-c790ce6b655d" (UID: "b8af5ae3-5412-49ee-b1c6-c790ce6b655d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:15.551325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.551295 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:15.631824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.631784 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:15.631824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:15.631824 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjjwv\" (UniqueName: \"kubernetes.io/projected/b8af5ae3-5412-49ee-b1c6-c790ce6b655d-kube-api-access-hjjwv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:16.083567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.083535 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:16.275315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.275273 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:16.288750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.288715 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:16.297950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.297918 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" event={"ID":"b8af5ae3-5412-49ee-b1c6-c790ce6b655d","Type":"ContainerDied","Data":"f8b167f950e5176c630e1b98e854f75e0585923843033b9f92965f8b14d13f0a"} Nov 01 23:02:16.298141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.297970 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b167f950e5176c630e1b98e854f75e0585923843033b9f92965f8b14d13f0a" Nov 01 23:02:16.298141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.297925 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4" Nov 01 23:02:16.313856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.313828 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:02:16.913060 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:16.913020 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:17.650776 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:17.650726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:18.472504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:18.472473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:19.269638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:19.269570 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:20.038066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:20.038027 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:21.078726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:21.078695 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:21.526784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:21.526749 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:22.244609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.244551 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:22.876629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.876592 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls"] Nov 01 23:02:22.877337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.877322 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b8af5ae3-5412-49ee-b1c6-c790ce6b655d" containerName="revoke-keys" Nov 01 23:02:22.877385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.877339 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8af5ae3-5412-49ee-b1c6-c790ce6b655d" containerName="revoke-keys" Nov 01 23:02:22.877473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.877464 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8af5ae3-5412-49ee-b1c6-c790ce6b655d" containerName="revoke-keys" Nov 01 23:02:22.881069 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.881051 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:22.885829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.885728 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:22.885829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.885774 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:02:22.891376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.891313 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls"] Nov 01 23:02:22.953898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.953848 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:22.954075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.954009 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvbx\" (UniqueName: \"kubernetes.io/projected/b680877d-2b1e-47f7-a7fb-65cde573777d-kube-api-access-vkvbx\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:22.954239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:22.954216 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.040886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.040849 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:23.055591 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.055545 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.055770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.055679 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvbx\" (UniqueName: \"kubernetes.io/projected/b680877d-2b1e-47f7-a7fb-65cde573777d-kube-api-access-vkvbx\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.055890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.055875 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.056481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.056455 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.057972 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.057949 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.063668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.063647 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvbx\" (UniqueName: \"kubernetes.io/projected/b680877d-2b1e-47f7-a7fb-65cde573777d-kube-api-access-vkvbx\") pod \"maas-api-revoke-sub-be5ac508-nd6ls\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.202117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.202079 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:23.484184 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:02:23.484152 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb680877d_2b1e_47f7_a7fb_65cde573777d.slice/crio-9ad1e89c9cf3ebf7d107c684250b8c1948d652b03a5b4810638fa3af75756bde WatchSource:0}: Error finding container 9ad1e89c9cf3ebf7d107c684250b8c1948d652b03a5b4810638fa3af75756bde: Status 404 returned error can't find the container with id 9ad1e89c9cf3ebf7d107c684250b8c1948d652b03a5b4810638fa3af75756bde Nov 01 23:02:23.484655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.484621 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls"] Nov 01 23:02:23.771414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:23.771332 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:24.298242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:24.298215 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:24.446599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:24.444407 4981 generic.go:356] "Generic (PLEG): container finished" podID="b680877d-2b1e-47f7-a7fb-65cde573777d" containerID="d8a5790fc877e94ceb6f5e1d9868da34777a1a2a6f83cb0306231c4f90d00600" exitCode=0 Nov 01 23:02:24.446599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:24.444501 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" event={"ID":"b680877d-2b1e-47f7-a7fb-65cde573777d","Type":"ContainerDied","Data":"d8a5790fc877e94ceb6f5e1d9868da34777a1a2a6f83cb0306231c4f90d00600"} Nov 01 23:02:24.446599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:24.444557 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" event={"ID":"b680877d-2b1e-47f7-a7fb-65cde573777d","Type":"ContainerStarted","Data":"9ad1e89c9cf3ebf7d107c684250b8c1948d652b03a5b4810638fa3af75756bde"} Nov 01 23:02:24.720955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:24.720892 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:25.460654 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.460622 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:25.646027 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.646003 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:25.811007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.810926 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-openshift-ca\") pod \"b680877d-2b1e-47f7-a7fb-65cde573777d\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " Nov 01 23:02:25.811491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.811047 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b680877d-2b1e-47f7-a7fb-65cde573777d-kube-api-access-vkvbx\" (UniqueName: \"kubernetes.io/projected/b680877d-2b1e-47f7-a7fb-65cde573777d-kube-api-access-vkvbx\") pod \"b680877d-2b1e-47f7-a7fb-65cde573777d\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " Nov 01 23:02:25.811491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.811205 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-xks-ca\") pod \"b680877d-2b1e-47f7-a7fb-65cde573777d\" (UID: \"b680877d-2b1e-47f7-a7fb-65cde573777d\") " Nov 01 23:02:25.811491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.811353 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-openshift-ca" pod "b680877d-2b1e-47f7-a7fb-65cde573777d" (UID: "b680877d-2b1e-47f7-a7fb-65cde573777d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:25.811926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.811908 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.813270 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.813246 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-xks-ca" pod "b680877d-2b1e-47f7-a7fb-65cde573777d" (UID: "b680877d-2b1e-47f7-a7fb-65cde573777d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:25.813371 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.813246 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b680877d-2b1e-47f7-a7fb-65cde573777d-kube-api-access-vkvbx" pod "b680877d-2b1e-47f7-a7fb-65cde573777d" (UID: "b680877d-2b1e-47f7-a7fb-65cde573777d"). InnerVolumeSpecName "kube-api-access-vkvbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:25.912921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.912888 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkvbx\" (UniqueName: \"kubernetes.io/projected/b680877d-2b1e-47f7-a7fb-65cde573777d-kube-api-access-vkvbx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:25.912921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:25.912920 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b680877d-2b1e-47f7-a7fb-65cde573777d-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:26.156113 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:26.156019 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:26.462952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:26.462923 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" Nov 01 23:02:26.463123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:26.462955 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls" event={"ID":"b680877d-2b1e-47f7-a7fb-65cde573777d","Type":"ContainerDied","Data":"9ad1e89c9cf3ebf7d107c684250b8c1948d652b03a5b4810638fa3af75756bde"} Nov 01 23:02:26.463123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:26.463004 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad1e89c9cf3ebf7d107c684250b8c1948d652b03a5b4810638fa3af75756bde" Nov 01 23:02:26.951545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:26.951504 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:27.452013 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.451975 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4"] Nov 01 23:02:27.452838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.452815 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b680877d-2b1e-47f7-a7fb-65cde573777d" containerName="revoke-keys" Nov 01 23:02:27.452838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.452837 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b680877d-2b1e-47f7-a7fb-65cde573777d" containerName="revoke-keys" Nov 01 23:02:27.453101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.453086 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b680877d-2b1e-47f7-a7fb-65cde573777d" containerName="revoke-keys" Nov 01 23:02:27.456877 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.456858 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.461028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.461010 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ee07ad-openshift-default-dockercfg-krtk4\"" Nov 01 23:02:27.471182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.471156 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4"] Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544361 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-certs\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544436 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf77v\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-kube-api-access-mf77v\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544691 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-istio-token\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544774 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544821 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7885a556-eb5d-479e-9744-7607d03fc3af-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544924 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-envoy\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544953 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-credential-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.544998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.544987 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-data\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.545555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.545104 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7885a556-eb5d-479e-9744-7607d03fc3af-istio-podinfo\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.579839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.579808 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:27.655459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.649754 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-certs\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.655856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.655830 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf77v\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-kube-api-access-mf77v\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.656395 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.656369 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-istio-token\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.667084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.667046 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.667784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.667759 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7885a556-eb5d-479e-9744-7607d03fc3af-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.682393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.682361 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-envoy\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.683004 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.682980 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-credential-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.683512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.683493 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-data\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.684049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.667687 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.684182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.683418 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-credential-socket\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.684286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.654039 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-certs\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.684371 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.672510 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7885a556-eb5d-479e-9744-7607d03fc3af-istiod-ca-cert\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.684473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.683943 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-data\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.684843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.684751 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7885a556-eb5d-479e-9744-7607d03fc3af-istio-podinfo\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.689898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.689828 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-envoy\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.696052 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.695728 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf77v\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-kube-api-access-mf77v\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.697161 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.696975 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-istio-token\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.712705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.702373 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7885a556-eb5d-479e-9744-7607d03fc3af-istio-podinfo\") pod \"e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.773491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.773456 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:27.970242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:27.970109 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:28.067054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.067024 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4"] Nov 01 23:02:28.067224 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:02:28.067073 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7885a556_eb5d_479e_9744_7607d03fc3af.slice/crio-276a94711219e34da664d84e453ec8e8a0795cb7d6c53e2ebb1917cf5e2874bd WatchSource:0}: Error finding container 276a94711219e34da664d84e453ec8e8a0795cb7d6c53e2ebb1917cf5e2874bd: Status 404 returned error can't find the container with id 276a94711219e34da664d84e453ec8e8a0795cb7d6c53e2ebb1917cf5e2874bd Nov 01 23:02:28.069794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.069760 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:02:28.069941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.069837 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:02:28.069941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.069895 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:02:28.324601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.324510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:28.484746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.484711 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" event={"ID":"7885a556-eb5d-479e-9744-7607d03fc3af","Type":"ContainerStarted","Data":"3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d"} Nov 01 23:02:28.484917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.484765 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" event={"ID":"7885a556-eb5d-479e-9744-7607d03fc3af","Type":"ContainerStarted","Data":"276a94711219e34da664d84e453ec8e8a0795cb7d6c53e2ebb1917cf5e2874bd"} Nov 01 23:02:28.514320 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.514268 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" podStartSLOduration=1.514253879 podStartE2EDuration="1.514253879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:28.513268458 +0000 UTC m=+2313.941146401" watchObservedRunningTime="2026-11-01 23:02:28.514253879 +0000 UTC m=+2313.942131810" Nov 01 23:02:28.626648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.626478 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:28.880652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.874535 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:28.998646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:28.991600 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:29.096598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:29.093277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:29.780399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:29.780031 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 01 23:02:29.780399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:29.780099 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 01 23:02:29.952851 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:29.952810 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:30.670887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:30.670848 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:30.777991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:30.777951 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 01 23:02:30.778195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:30.778015 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 01 23:02:31.552329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:31.552295 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:31.765108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:31.765074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:31.783850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:31.783007 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 01 23:02:31.783850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:31.783075 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 01 23:02:31.869801 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:31.869705 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:32.563262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:32.563037 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:32.796612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:32.795719 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 01 23:02:32.796612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:32.795783 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 01 23:02:33.063369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.063330 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:33.191349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.191273 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:33.206699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.200709 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx"] Nov 01 23:02:33.208352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.208328 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:33.208992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.208959 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx"] Nov 01 23:02:33.447598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.437282 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-337dee\"" Nov 01 23:02:33.447598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.437428 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-337dee-dockercfg-8b4k8\"" Nov 01 23:02:33.525080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.525051 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a184b51d-c4c0-449d-bd87-dcf50c02c65a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:33.525239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.525098 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf959\" (UniqueName: \"kubernetes.io/projected/a184b51d-c4c0-449d-bd87-dcf50c02c65a-kube-api-access-xf959\") pod \"payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:33.626859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.626817 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a184b51d-c4c0-449d-bd87-dcf50c02c65a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:33.627098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.626878 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf959\" (UniqueName: \"kubernetes.io/projected/a184b51d-c4c0-449d-bd87-dcf50c02c65a-kube-api-access-xf959\") pod \"payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:33.627657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.627636 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a184b51d-c4c0-449d-bd87-dcf50c02c65a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:33.647290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.647258 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf959\" (UniqueName: \"kubernetes.io/projected/a184b51d-c4c0-449d-bd87-dcf50c02c65a-kube-api-access-xf959\") pod \"payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:33.741647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.739367 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:33.796611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.781918 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 01 23:02:33.796611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.781991 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 01 23:02:33.819124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:33.819095 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:34.135280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.135253 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx"] Nov 01 23:02:34.135402 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:02:34.135376 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda184b51d_c4c0_449d_bd87_dcf50c02c65a.slice/crio-8030487bb3c4e01b288b6236d1b0b89f6e3f41ac87b0069001376985892b8240 WatchSource:0}: Error finding container 8030487bb3c4e01b288b6236d1b0b89f6e3f41ac87b0069001376985892b8240: Status 404 returned error can't find the container with id 8030487bb3c4e01b288b6236d1b0b89f6e3f41ac87b0069001376985892b8240 Nov 01 23:02:34.513027 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.512980 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:34.581957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.581911 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" event={"ID":"a184b51d-c4c0-449d-bd87-dcf50c02c65a","Type":"ContainerStarted","Data":"c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478"} Nov 01 23:02:34.582162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.581990 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" event={"ID":"a184b51d-c4c0-449d-bd87-dcf50c02c65a","Type":"ContainerStarted","Data":"8030487bb3c4e01b288b6236d1b0b89f6e3f41ac87b0069001376985892b8240"} Nov 01 23:02:34.582365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.582347 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:34.605451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.605390 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" podStartSLOduration=1.6053708549999999 podStartE2EDuration="1.605370855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:34.599177624 +0000 UTC m=+2320.027055556" watchObservedRunningTime="2026-11-01 23:02:34.605370855 +0000 UTC m=+2320.033248786" Nov 01 23:02:34.779962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.779885 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:34.780359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.780348 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:34.781049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:34.781030 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:02:35.448948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:35.448906 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:36.152456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:36.152419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:36.870499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:36.870463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:37.027554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.027515 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz"] Nov 01 23:02:37.031961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.031929 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1106dd29-cwdvz"] Nov 01 23:02:37.120897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.120810 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:37.539101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.539069 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2"] Nov 01 23:02:37.543787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.543767 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.548303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.548279 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:37.548541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.548524 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:02:37.552669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.552649 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2"] Nov 01 23:02:37.600945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.600910 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.601153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.601044 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvrx6\" (UniqueName: \"kubernetes.io/projected/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-kube-api-access-wvrx6\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.601153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.601115 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.703139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.703102 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvrx6\" (UniqueName: \"kubernetes.io/projected/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-kube-api-access-wvrx6\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.703303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.703194 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.703370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.703357 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.704034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.704006 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.705707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.705686 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.712997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.712977 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvrx6\" (UniqueName: \"kubernetes.io/projected/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-kube-api-access-wvrx6\") pod \"maas-api-revoke-sub-0f1d9592-99qc2\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:37.855089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.855010 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:37.861619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:37.861597 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:38.186928 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:02:38.186896 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9b8edf_a3c3_432f_bc7a_9e9ba7fb32fe.slice/crio-6a5dd05d2a3d37aaab54a0c51e31ee5bcb53aa76b381b6537f551476dbd4ad2f WatchSource:0}: Error finding container 6a5dd05d2a3d37aaab54a0c51e31ee5bcb53aa76b381b6537f551476dbd4ad2f: Status 404 returned error can't find the container with id 6a5dd05d2a3d37aaab54a0c51e31ee5bcb53aa76b381b6537f551476dbd4ad2f Nov 01 23:02:38.187637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:38.187615 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2"] Nov 01 23:02:38.618674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:38.618637 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" event={"ID":"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe","Type":"ContainerStarted","Data":"67b476b406db66566269c9e448ea859eca9eedbedab1c0d5c07320034eabb139"} Nov 01 23:02:38.618855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:38.618702 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" event={"ID":"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe","Type":"ContainerStarted","Data":"6a5dd05d2a3d37aaab54a0c51e31ee5bcb53aa76b381b6537f551476dbd4ad2f"} Nov 01 23:02:38.639533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:38.639481 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" podStartSLOduration=1.6394680130000001 podStartE2EDuration="1.639468013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:38.63467999 +0000 UTC m=+2324.062557922" watchObservedRunningTime="2026-11-01 23:02:38.639468013 +0000 UTC m=+2324.067345945" Nov 01 23:02:38.657307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:38.657273 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:38.757953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:38.757816 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c" path="/var/lib/kubelet/pods/92cc3a9c-1a8a-49ab-b7b6-a5f8ba0d6b6c/volumes" Nov 01 23:02:39.414963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:39.414880 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:39.628337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:39.628304 4981 generic.go:356] "Generic (PLEG): container finished" podID="2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" containerID="67b476b406db66566269c9e448ea859eca9eedbedab1c0d5c07320034eabb139" exitCode=0 Nov 01 23:02:39.628516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:39.628383 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" event={"ID":"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe","Type":"ContainerDied","Data":"67b476b406db66566269c9e448ea859eca9eedbedab1c0d5c07320034eabb139"} Nov 01 23:02:40.260356 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.260307 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:40.805606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.805543 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:40.871862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.871825 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-openshift-ca\") pod \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " Nov 01 23:02:40.872066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.871896 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-kube-api-access-wvrx6\" (UniqueName: \"kubernetes.io/projected/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-kube-api-access-wvrx6\") pod \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " Nov 01 23:02:40.872256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.872227 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-openshift-ca" pod "2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" (UID: "2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:40.872358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.872243 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-xks-ca\") pod \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\" (UID: \"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe\") " Nov 01 23:02:40.873102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.873073 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:40.874678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.874642 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-kube-api-access-wvrx6" pod "2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" (UID: "2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe"). InnerVolumeSpecName "kube-api-access-wvrx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:40.874790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.874680 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-xks-ca" pod "2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" (UID: "2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:40.974813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.974783 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:40.974813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:40.974810 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvrx6\" (UniqueName: \"kubernetes.io/projected/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe-kube-api-access-wvrx6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:41.096714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:41.096677 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:41.646559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:41.646532 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" Nov 01 23:02:41.646817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:41.646564 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2" event={"ID":"2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe","Type":"ContainerDied","Data":"6a5dd05d2a3d37aaab54a0c51e31ee5bcb53aa76b381b6537f551476dbd4ad2f"} Nov 01 23:02:41.646817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:41.646651 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a5dd05d2a3d37aaab54a0c51e31ee5bcb53aa76b381b6537f551476dbd4ad2f" Nov 01 23:02:41.930035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:41.930007 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:42.843663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:42.843613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:43.471962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:43.471921 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:44.177955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:44.177914 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:44.527949 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:02:44.527862 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:02:44.952805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:44.952765 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:45.593386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:45.593351 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:02:45.775195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:45.775162 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:46.458513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:46.458482 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:47.172738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:47.172703 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:48.093726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:48.093693 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:48.820048 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:48.820017 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:48.921667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:48.921607 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:49.069073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.069040 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5"] Nov 01 23:02:49.073185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.073127 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be8b244a-vtkl5"] Nov 01 23:02:49.112007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.111973 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:49.246034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.245995 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h"] Nov 01 23:02:49.248068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.248040 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" containerName="revoke-keys" Nov 01 23:02:49.248068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.248064 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" containerName="revoke-keys" Nov 01 23:02:49.248260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.248236 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" containerName="revoke-keys" Nov 01 23:02:49.404658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.398733 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.412420 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.412378 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:02:49.452163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.441400 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:02:49.468946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.468908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7z25\" (UniqueName: \"kubernetes.io/projected/c348a862-625d-466e-b74e-9c7aef5a3ff1-kube-api-access-t7z25\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.469313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.469288 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.469666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.469639 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.479981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.479950 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h"] Nov 01 23:02:49.571524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.571485 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7z25\" (UniqueName: \"kubernetes.io/projected/c348a862-625d-466e-b74e-9c7aef5a3ff1-kube-api-access-t7z25\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.571782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.571657 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.572076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.571917 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.573066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.573038 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.574747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.574704 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.581156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.581037 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7z25\" (UniqueName: \"kubernetes.io/projected/c348a862-625d-466e-b74e-9c7aef5a3ff1-kube-api-access-t7z25\") pod \"maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:49.782600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:49.780390 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:50.107067 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:50.107038 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h"] Nov 01 23:02:50.107548 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:02:50.107524 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc348a862_625d_466e_b74e_9c7aef5a3ff1.slice/crio-823d65c6cf7b9d3eae0927f452614580d8fff67da72a57df9b01392bc63d6ae0 WatchSource:0}: Error finding container 823d65c6cf7b9d3eae0927f452614580d8fff67da72a57df9b01392bc63d6ae0: Status 404 returned error can't find the container with id 823d65c6cf7b9d3eae0927f452614580d8fff67da72a57df9b01392bc63d6ae0 Nov 01 23:02:50.340249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:50.340216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:50.736705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:50.736623 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6185c2-0df8-46e9-aabc-43f55718935e" path="/var/lib/kubelet/pods/2a6185c2-0df8-46e9-aabc-43f55718935e/volumes" Nov 01 23:02:50.736993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:50.736976 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" event={"ID":"c348a862-625d-466e-b74e-9c7aef5a3ff1","Type":"ContainerStarted","Data":"7993f2e7dd705a8e79357c3086c11174540b7400b528528c830e4356d680059f"} Nov 01 23:02:50.737058 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:50.737019 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" event={"ID":"c348a862-625d-466e-b74e-9c7aef5a3ff1","Type":"ContainerStarted","Data":"823d65c6cf7b9d3eae0927f452614580d8fff67da72a57df9b01392bc63d6ae0"} Nov 01 23:02:50.771014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:50.770961 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" podStartSLOduration=1.77094749 podStartE2EDuration="1.77094749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:50.766756638 +0000 UTC m=+2336.194634569" watchObservedRunningTime="2026-11-01 23:02:50.77094749 +0000 UTC m=+2336.198825421" Nov 01 23:02:51.037276 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:02:51.029386 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc348a862_625d_466e_b74e_9c7aef5a3ff1.slice/crio-conmon-7993f2e7dd705a8e79357c3086c11174540b7400b528528c830e4356d680059f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc348a862_625d_466e_b74e_9c7aef5a3ff1.slice/crio-7993f2e7dd705a8e79357c3086c11174540b7400b528528c830e4356d680059f.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:02:51.048612 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:02:51.041925 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc348a862_625d_466e_b74e_9c7aef5a3ff1.slice/crio-7993f2e7dd705a8e79357c3086c11174540b7400b528528c830e4356d680059f.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:02:51.741729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:51.741697 4981 generic.go:356] "Generic (PLEG): container finished" podID="c348a862-625d-466e-b74e-9c7aef5a3ff1" containerID="7993f2e7dd705a8e79357c3086c11174540b7400b528528c830e4356d680059f" exitCode=0 Nov 01 23:02:51.742270 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:51.741775 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" event={"ID":"c348a862-625d-466e-b74e-9c7aef5a3ff1","Type":"ContainerDied","Data":"7993f2e7dd705a8e79357c3086c11174540b7400b528528c830e4356d680059f"} Nov 01 23:02:51.763218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:51.763190 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:52.654142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.654114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:52.930804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.930779 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:52.952790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.952765 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-xks-ca\") pod \"c348a862-625d-466e-b74e-9c7aef5a3ff1\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " Nov 01 23:02:52.952951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.952804 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-openshift-ca\") pod \"c348a862-625d-466e-b74e-9c7aef5a3ff1\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " Nov 01 23:02:52.953180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.953145 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-openshift-ca" pod "c348a862-625d-466e-b74e-9c7aef5a3ff1" (UID: "c348a862-625d-466e-b74e-9c7aef5a3ff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:02:52.953180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.953157 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c348a862-625d-466e-b74e-9c7aef5a3ff1-kube-api-access-t7z25\" (UniqueName: \"kubernetes.io/projected/c348a862-625d-466e-b74e-9c7aef5a3ff1-kube-api-access-t7z25\") pod \"c348a862-625d-466e-b74e-9c7aef5a3ff1\" (UID: \"c348a862-625d-466e-b74e-9c7aef5a3ff1\") " Nov 01 23:02:52.953871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.953842 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:52.955061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.955035 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-xks-ca" pod "c348a862-625d-466e-b74e-9c7aef5a3ff1" (UID: "c348a862-625d-466e-b74e-9c7aef5a3ff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:02:52.955176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:52.955151 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c348a862-625d-466e-b74e-9c7aef5a3ff1-kube-api-access-t7z25" pod "c348a862-625d-466e-b74e-9c7aef5a3ff1" (UID: "c348a862-625d-466e-b74e-9c7aef5a3ff1"). InnerVolumeSpecName "kube-api-access-t7z25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:02:53.055818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:53.055786 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c348a862-625d-466e-b74e-9c7aef5a3ff1-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:53.055818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:53.055813 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7z25\" (UniqueName: \"kubernetes.io/projected/c348a862-625d-466e-b74e-9c7aef5a3ff1-kube-api-access-t7z25\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:02:53.477107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:53.477076 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:53.760765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:53.760674 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" event={"ID":"c348a862-625d-466e-b74e-9c7aef5a3ff1","Type":"ContainerDied","Data":"823d65c6cf7b9d3eae0927f452614580d8fff67da72a57df9b01392bc63d6ae0"} Nov 01 23:02:53.760765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:53.760694 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h" Nov 01 23:02:53.760765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:53.760720 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="823d65c6cf7b9d3eae0927f452614580d8fff67da72a57df9b01392bc63d6ae0" Nov 01 23:02:54.276643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:54.276558 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:55.334888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:55.334779 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:56.030938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:56.030882 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:56.879092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:56.879053 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:57.766165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:57.766132 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:58.544163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:58.544129 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:59.264516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:59.264476 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:59.409764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:59.409479 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9"] Nov 01 23:02:59.420803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:59.420770 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk"] Nov 01 23:02:59.471296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:59.471258 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:02:59.572627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:02:59.572554 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:00.180799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:00.180768 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:00.970033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:00.970003 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:01.654660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:01.654627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:01.797267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:01.797232 4981 scope.go:122] "RemoveContainer" containerID="a3b2236310aa1469bdda2fb00545fbddd54de2ae87a6150fabb082b09199be5a" Nov 01 23:03:01.807927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:01.807902 4981 scope.go:122] "RemoveContainer" containerID="ab9ca7d9dc39474541c04a5894133198ddce35a706cbc345ba87c943bb9f9ef1" Nov 01 23:03:02.346206 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:02.346175 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:03.154324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:03.154288 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:03.968356 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:03.968308 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:04.410382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.410292 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" podUID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" containerName="payload-processing" containerID="cri-o://09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f" gracePeriod=25 Nov 01 23:03:04.421621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.421563 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" podUID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" containerName="payload-pre-processing" containerID="cri-o://96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea" gracePeriod=25 Nov 01 23:03:04.534170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.534133 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z"] Nov 01 23:03:04.534447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.534422 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" podUID="d4f62aab-95cf-4fb9-86f9-89534b73e1b9" containerName="maas-api" containerID="cri-o://0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b" gracePeriod=30 Nov 01 23:03:04.585150 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.585120 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:04.762598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.762430 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:04.892635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.892604 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:04.958368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.958302 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h"] Nov 01 23:03:04.963315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:04.963289 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c49240-vlj6h"] Nov 01 23:03:05.006368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.006336 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:05.094390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.094364 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 23:03:05.184756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.184726 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-kube-api-access-zhp22\" (UniqueName: \"kubernetes.io/projected/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-kube-api-access-zhp22\") pod \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " Nov 01 23:03:05.184968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.184900 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-openshift-service-ca\") pod \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " Nov 01 23:03:05.184968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.184930 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-metrics-certs\") pod \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " Nov 01 23:03:05.185104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.184976 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-maas-api-tls\") pod \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\" (UID: \"d4f62aab-95cf-4fb9-86f9-89534b73e1b9\") " Nov 01 23:03:05.185348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.185316 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-openshift-service-ca" pod "d4f62aab-95cf-4fb9-86f9-89534b73e1b9" (UID: "d4f62aab-95cf-4fb9-86f9-89534b73e1b9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:05.185638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.185616 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:05.187283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.187261 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-maas-api-tls" pod "d4f62aab-95cf-4fb9-86f9-89534b73e1b9" (UID: "d4f62aab-95cf-4fb9-86f9-89534b73e1b9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:03:05.187511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.187490 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-metrics-certs" pod "d4f62aab-95cf-4fb9-86f9-89534b73e1b9" (UID: "d4f62aab-95cf-4fb9-86f9-89534b73e1b9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:03:05.187617 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.187556 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-kube-api-access-zhp22" pod "d4f62aab-95cf-4fb9-86f9-89534b73e1b9" (UID: "d4f62aab-95cf-4fb9-86f9-89534b73e1b9"). InnerVolumeSpecName "kube-api-access-zhp22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:05.287306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.287275 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhp22\" (UniqueName: \"kubernetes.io/projected/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-kube-api-access-zhp22\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:05.287306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.287304 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:05.287511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.287315 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4f62aab-95cf-4fb9-86f9-89534b73e1b9-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:05.360235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.360155 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:05.892880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.892845 4981 generic.go:356] "Generic (PLEG): container finished" podID="d4f62aab-95cf-4fb9-86f9-89534b73e1b9" containerID="0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b" exitCode=0 Nov 01 23:03:05.893088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.892895 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" Nov 01 23:03:05.893088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.892898 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" event={"ID":"d4f62aab-95cf-4fb9-86f9-89534b73e1b9","Type":"ContainerDied","Data":"0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b"} Nov 01 23:03:05.893088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.892950 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z" event={"ID":"d4f62aab-95cf-4fb9-86f9-89534b73e1b9","Type":"ContainerDied","Data":"90d80d9933eca42d0b7fc31c124f2ca2b04a43e05d3c6fc0a9cd7a0dea80a0de"} Nov 01 23:03:05.893088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.892980 4981 scope.go:122] "RemoveContainer" containerID="0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b" Nov 01 23:03:05.905548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.905525 4981 scope.go:122] "RemoveContainer" containerID="0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b" Nov 01 23:03:05.906524 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:03:05.905885 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b\": container with ID starting with 0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b not found: ID does not exist" containerID="0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b" Nov 01 23:03:05.906524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.905921 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b"} err="failed to get container status \"0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b\": rpc error: code = NotFound desc = could not find container \"0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b\": container with ID starting with 0c9fa9bbd485555a0ee5c4e596ca7e3103bcfa08829cefdb1aeaf9eeeecd257b not found: ID does not exist" Nov 01 23:03:05.919047 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.919023 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z"] Nov 01 23:03:05.925364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.925342 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c49240-755c9676fb-vvt2z"] Nov 01 23:03:05.967257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:05.967233 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:06.695191 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:06.695154 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:06.747038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:06.746994 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c348a862-625d-466e-b74e-9c7aef5a3ff1" path="/var/lib/kubelet/pods/c348a862-625d-466e-b74e-9c7aef5a3ff1/volumes" Nov 01 23:03:06.747877 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:06.747851 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f62aab-95cf-4fb9-86f9-89534b73e1b9" path="/var/lib/kubelet/pods/d4f62aab-95cf-4fb9-86f9-89534b73e1b9/volumes" Nov 01 23:03:07.043987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.043903 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst"] Nov 01 23:03:07.044626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.044611 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c348a862-625d-466e-b74e-9c7aef5a3ff1" containerName="revoke-keys" Nov 01 23:03:07.044674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.044627 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c348a862-625d-466e-b74e-9c7aef5a3ff1" containerName="revoke-keys" Nov 01 23:03:07.044674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.044666 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d4f62aab-95cf-4fb9-86f9-89534b73e1b9" containerName="maas-api" Nov 01 23:03:07.044674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.044671 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f62aab-95cf-4fb9-86f9-89534b73e1b9" containerName="maas-api" Nov 01 23:03:07.044842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.044801 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c348a862-625d-466e-b74e-9c7aef5a3ff1" containerName="revoke-keys" Nov 01 23:03:07.044842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.044816 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f62aab-95cf-4fb9-86f9-89534b73e1b9" containerName="maas-api" Nov 01 23:03:07.053324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.053299 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.059735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.059710 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-7366bf-openshift-default-dockercfg-9nnps\"" Nov 01 23:03:07.061070 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.061003 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst"] Nov 01 23:03:07.216959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.216925 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.217417 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.217392 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.217701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.217681 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563155b-37e1-4023-a721-545a505f7f6f-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.217907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.217890 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.218113 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.218096 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.218351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.218317 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq2sx\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-kube-api-access-cq2sx\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.218670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.218652 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563155b-37e1-4023-a721-545a505f7f6f-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.218887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.218872 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.219103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.219086 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.320723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.320652 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563155b-37e1-4023-a721-545a505f7f6f-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.320723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.320716 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.320905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.320736 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.320905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.320784 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.320905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.320878 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.320916 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563155b-37e1-4023-a721-545a505f7f6f-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321104 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321162 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321167 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321241 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq2sx\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-kube-api-access-cq2sx\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321412 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563155b-37e1-4023-a721-545a505f7f6f-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321554 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321690 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321621 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.321794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.321777 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.323538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.323513 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563155b-37e1-4023-a721-545a505f7f6f-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.323669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.323566 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.329749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.329726 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.330161 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.330143 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq2sx\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-kube-api-access-cq2sx\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.373457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.373428 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:07.531234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.531200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:07.661092 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:03:07.661053 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1563155b_37e1_4023_a721_545a505f7f6f.slice/crio-60e1ab35b49b69891036be4ff4ec2fa4f79014b4eed8d74b59086778b8bbccc0 WatchSource:0}: Error finding container 60e1ab35b49b69891036be4ff4ec2fa4f79014b4eed8d74b59086778b8bbccc0: Status 404 returned error can't find the container with id 60e1ab35b49b69891036be4ff4ec2fa4f79014b4eed8d74b59086778b8bbccc0 Nov 01 23:03:07.661538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.661510 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst"] Nov 01 23:03:07.664242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.664200 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:03:07.664349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.664295 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:03:07.664349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.664325 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:03:07.914351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.914262 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" event={"ID":"1563155b-37e1-4023-a721-545a505f7f6f","Type":"ContainerStarted","Data":"7898651c56700954f3e68795eb8d8eee2693cab2b6c3a2c8528c089611e25d07"} Nov 01 23:03:07.914351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.914319 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" event={"ID":"1563155b-37e1-4023-a721-545a505f7f6f","Type":"ContainerStarted","Data":"60e1ab35b49b69891036be4ff4ec2fa4f79014b4eed8d74b59086778b8bbccc0"} Nov 01 23:03:07.936468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:07.936406 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podStartSLOduration=0.936392043 podStartE2EDuration="936.392043ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:07.932684513 +0000 UTC m=+2353.360562446" watchObservedRunningTime="2026-11-01 23:03:07.936392043 +0000 UTC m=+2353.364269974" Nov 01 23:03:08.247300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:08.247272 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:08.374295 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:08.374255 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:08.376253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:08.376218 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 23:03:08.376467 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:08.376283 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 23:03:09.215505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:09.215472 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:09.394602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:09.383008 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 23:03:09.394602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:09.383069 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 23:03:09.900654 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:09.900618 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:10.394599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:10.391953 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 23:03:10.394599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:10.392010 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 23:03:10.770601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:10.766725 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" podUID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:03:10.770601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:10.767081 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" podUID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:03:10.852434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:10.852401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:11.398804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:11.398756 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 23:03:11.399289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:11.398828 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 23:03:11.668456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:11.668367 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:11.861659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:11.861625 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:11.966877 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:11.966847 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:12.199250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.199209 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:12.256649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.247638 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx"] Nov 01 23:03:12.501822 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.482103 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:12.529650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.521076 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 23:03:12.529650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.521190 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 23:03:12.566338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.566303 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:12.614089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.612740 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45"] Nov 01 23:03:12.621019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.620984 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45"] Nov 01 23:03:12.621170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.621082 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.639285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.639249 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:03:12.639415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.639255 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:03:12.663950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.663918 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:12.733609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.731560 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.733609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.731708 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.733609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.731742 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94j2j\" (UniqueName: \"kubernetes.io/projected/f0bfac70-72e1-450c-9b18-3137a73cedef-kube-api-access-94j2j\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.839646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.835812 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.839646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.835960 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.839646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.835996 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94j2j\" (UniqueName: \"kubernetes.io/projected/f0bfac70-72e1-450c-9b18-3137a73cedef-kube-api-access-94j2j\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.839646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.837379 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.865599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.858323 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.865599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.861627 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94j2j\" (UniqueName: \"kubernetes.io/projected/f0bfac70-72e1-450c-9b18-3137a73cedef-kube-api-access-94j2j\") pod \"maas-api-revoke-keys-e2e-inf-b-64b196-zfp45\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:12.941630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:12.941538 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:13.250290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:13.245705 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45"] Nov 01 23:03:13.255392 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:03:13.255334 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bfac70_72e1_450c_9b18_3137a73cedef.slice/crio-54f3655f0877151908d2af197131a606d19aa96ef5f693bf657a9a4e1f6e0576 WatchSource:0}: Error finding container 54f3655f0877151908d2af197131a606d19aa96ef5f693bf657a9a4e1f6e0576: Status 404 returned error can't find the container with id 54f3655f0877151908d2af197131a606d19aa96ef5f693bf657a9a4e1f6e0576 Nov 01 23:03:13.395600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:13.390898 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 23:03:13.395600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:13.390971 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 23:03:13.459234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:13.459198 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:13.968073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:13.968041 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" event={"ID":"f0bfac70-72e1-450c-9b18-3137a73cedef","Type":"ContainerStarted","Data":"a1b67b5859878ee2618efba654fc11d4ad98b57fc0778e95c36cc5989434164f"} Nov 01 23:03:13.968511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:13.968095 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" event={"ID":"f0bfac70-72e1-450c-9b18-3137a73cedef","Type":"ContainerStarted","Data":"54f3655f0877151908d2af197131a606d19aa96ef5f693bf657a9a4e1f6e0576"} Nov 01 23:03:14.003062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:14.002990 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" podStartSLOduration=2.00296736 podStartE2EDuration="2.00296736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:14.001061089 +0000 UTC m=+2359.428939021" watchObservedRunningTime="2026-11-01 23:03:14.00296736 +0000 UTC m=+2359.430845291" Nov 01 23:03:14.344483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:14.344446 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:14.377815 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:14.377779 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 23:03:14.378068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:14.378044 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 23:03:14.531803 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:03:14.531765 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:03:14.925273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:14.925218 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:14.977963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:14.977929 4981 generic.go:356] "Generic (PLEG): container finished" podID="f0bfac70-72e1-450c-9b18-3137a73cedef" containerID="a1b67b5859878ee2618efba654fc11d4ad98b57fc0778e95c36cc5989434164f" exitCode=0 Nov 01 23:03:14.978406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:14.977979 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" event={"ID":"f0bfac70-72e1-450c-9b18-3137a73cedef","Type":"ContainerDied","Data":"a1b67b5859878ee2618efba654fc11d4ad98b57fc0778e95c36cc5989434164f"} Nov 01 23:03:15.025786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:15.025734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:15.378896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:15.378815 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:15.379289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:15.379276 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:15.380096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:15.380075 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:03:15.501391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:15.501352 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:15.804789 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:15.804756 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:16.146955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.146928 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:16.302010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.301973 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-openshift-ca\") pod \"f0bfac70-72e1-450c-9b18-3137a73cedef\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " Nov 01 23:03:16.302223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.302096 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-xks-ca\") pod \"f0bfac70-72e1-450c-9b18-3137a73cedef\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " Nov 01 23:03:16.302293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.302241 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bfac70-72e1-450c-9b18-3137a73cedef-kube-api-access-94j2j\" (UniqueName: \"kubernetes.io/projected/f0bfac70-72e1-450c-9b18-3137a73cedef-kube-api-access-94j2j\") pod \"f0bfac70-72e1-450c-9b18-3137a73cedef\" (UID: \"f0bfac70-72e1-450c-9b18-3137a73cedef\") " Nov 01 23:03:16.302480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.302455 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-openshift-ca" pod "f0bfac70-72e1-450c-9b18-3137a73cedef" (UID: "f0bfac70-72e1-450c-9b18-3137a73cedef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:16.303038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.303011 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:16.305414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.304971 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-xks-ca" pod "f0bfac70-72e1-450c-9b18-3137a73cedef" (UID: "f0bfac70-72e1-450c-9b18-3137a73cedef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:03:16.305414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.305071 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bfac70-72e1-450c-9b18-3137a73cedef-kube-api-access-94j2j" pod "f0bfac70-72e1-450c-9b18-3137a73cedef" (UID: "f0bfac70-72e1-450c-9b18-3137a73cedef"). InnerVolumeSpecName "kube-api-access-94j2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:16.404079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.404014 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfac70-72e1-450c-9b18-3137a73cedef-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:16.404079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.404036 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94j2j\" (UniqueName: \"kubernetes.io/projected/f0bfac70-72e1-450c-9b18-3137a73cedef-kube-api-access-94j2j\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:16.562229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.562192 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:16.995672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.995560 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" Nov 01 23:03:16.995672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.995598 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45" event={"ID":"f0bfac70-72e1-450c-9b18-3137a73cedef","Type":"ContainerDied","Data":"54f3655f0877151908d2af197131a606d19aa96ef5f693bf657a9a4e1f6e0576"} Nov 01 23:03:16.995672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:16.995648 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54f3655f0877151908d2af197131a606d19aa96ef5f693bf657a9a4e1f6e0576" Nov 01 23:03:17.202145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.202106 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:17.252656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.249272 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" podUID="a184b51d-c4c0-449d-bd87-dcf50c02c65a" containerName="payload-processing" containerID="cri-o://c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478" gracePeriod=25 Nov 01 23:03:17.555341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.555308 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:03:17.622076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.622045 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a184b51d-c4c0-449d-bd87-dcf50c02c65a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a184b51d-c4c0-449d-bd87-dcf50c02c65a-plugins-config-volume\") pod \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " Nov 01 23:03:17.622274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.622250 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a184b51d-c4c0-449d-bd87-dcf50c02c65a-kube-api-access-xf959\" (UniqueName: \"kubernetes.io/projected/a184b51d-c4c0-449d-bd87-dcf50c02c65a-kube-api-access-xf959\") pod \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\" (UID: \"a184b51d-c4c0-449d-bd87-dcf50c02c65a\") " Nov 01 23:03:17.622518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.622488 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a184b51d-c4c0-449d-bd87-dcf50c02c65a-plugins-config-volume" pod "a184b51d-c4c0-449d-bd87-dcf50c02c65a" (UID: "a184b51d-c4c0-449d-bd87-dcf50c02c65a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:17.622966 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.622944 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a184b51d-c4c0-449d-bd87-dcf50c02c65a-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.625165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.625135 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a184b51d-c4c0-449d-bd87-dcf50c02c65a-kube-api-access-xf959" pod "a184b51d-c4c0-449d-bd87-dcf50c02c65a" (UID: "a184b51d-c4c0-449d-bd87-dcf50c02c65a"). InnerVolumeSpecName "kube-api-access-xf959". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:17.724951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.724915 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf959\" (UniqueName: \"kubernetes.io/projected/a184b51d-c4c0-449d-bd87-dcf50c02c65a-kube-api-access-xf959\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:17.893130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:17.893022 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:18.004889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.004853 4981 generic.go:356] "Generic (PLEG): container finished" podID="a184b51d-c4c0-449d-bd87-dcf50c02c65a" containerID="c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478" exitCode=0 Nov 01 23:03:18.005085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.004900 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" Nov 01 23:03:18.005085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.004942 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" event={"ID":"a184b51d-c4c0-449d-bd87-dcf50c02c65a","Type":"ContainerDied","Data":"c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478"} Nov 01 23:03:18.005085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.005003 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx" event={"ID":"a184b51d-c4c0-449d-bd87-dcf50c02c65a","Type":"ContainerDied","Data":"8030487bb3c4e01b288b6236d1b0b89f6e3f41ac87b0069001376985892b8240"} Nov 01 23:03:18.005085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.005032 4981 scope.go:122] "RemoveContainer" containerID="c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478" Nov 01 23:03:18.019172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.019150 4981 scope.go:122] "RemoveContainer" containerID="c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478" Nov 01 23:03:18.019464 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:03:18.019437 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478\": container with ID starting with c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478 not found: ID does not exist" containerID="c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478" Nov 01 23:03:18.019568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.019469 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478"} err="failed to get container status \"c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478\": rpc error: code = NotFound desc = could not find container \"c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478\": container with ID starting with c0793c92800fd7d8228e2be3f658b8ec1ce984e961e3056c4be6f312fa458478 not found: ID does not exist" Nov 01 23:03:18.036145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.036120 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx"] Nov 01 23:03:18.040224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.040199 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-337dee-86c5d4db99-ghhnx"] Nov 01 23:03:18.575244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.575209 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:18.736981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:18.736953 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a184b51d-c4c0-449d-bd87-dcf50c02c65a" path="/var/lib/kubelet/pods/a184b51d-c4c0-449d-bd87-dcf50c02c65a/volumes" Nov 01 23:03:19.241560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:19.241526 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:19.443401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:19.443362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:19.973814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:19.973781 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:20.723547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:20.723518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:20.752886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:20.752848 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" podUID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:03:20.753035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:20.752848 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" podUID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:03:21.550257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:21.550226 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:22.169237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:22.169205 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:22.746229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:22.746201 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:22.847004 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:22.846972 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:22.953651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:22.953135 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:23.560123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:23.560090 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:24.241565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.241531 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:24.715141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.715116 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 23:03:24.744483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.744457 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 23:03:24.793545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.793450 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-plugins-config-volume\") pod \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " Nov 01 23:03:24.793769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.793567 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-kube-api-access-lgqd2\" (UniqueName: \"kubernetes.io/projected/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-kube-api-access-lgqd2\") pod \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\" (UID: \"2e75445e-7d5c-4ee1-86b7-9c0a6b874572\") " Nov 01 23:03:24.793996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.793965 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-plugins-config-volume" pod "2e75445e-7d5c-4ee1-86b7-9c0a6b874572" (UID: "2e75445e-7d5c-4ee1-86b7-9c0a6b874572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:24.794427 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.794402 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:24.795793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.795765 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-kube-api-access-lgqd2" pod "2e75445e-7d5c-4ee1-86b7-9c0a6b874572" (UID: "2e75445e-7d5c-4ee1-86b7-9c0a6b874572"). InnerVolumeSpecName "kube-api-access-lgqd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:24.895846 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.895813 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c38bb28-8f0c-4352-bc91-a259731f3c9a-kube-api-access-ptzns\" (UniqueName: \"kubernetes.io/projected/5c38bb28-8f0c-4352-bc91-a259731f3c9a-kube-api-access-ptzns\") pod \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " Nov 01 23:03:24.895993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.895865 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c38bb28-8f0c-4352-bc91-a259731f3c9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c38bb28-8f0c-4352-bc91-a259731f3c9a-plugins-config-volume\") pod \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\" (UID: \"5c38bb28-8f0c-4352-bc91-a259731f3c9a\") " Nov 01 23:03:24.896238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.896217 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c38bb28-8f0c-4352-bc91-a259731f3c9a-plugins-config-volume" pod "5c38bb28-8f0c-4352-bc91-a259731f3c9a" (UID: "5c38bb28-8f0c-4352-bc91-a259731f3c9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:24.896422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.896402 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c38bb28-8f0c-4352-bc91-a259731f3c9a-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:24.896462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.896429 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgqd2\" (UniqueName: \"kubernetes.io/projected/2e75445e-7d5c-4ee1-86b7-9c0a6b874572-kube-api-access-lgqd2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:24.898086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.898055 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c38bb28-8f0c-4352-bc91-a259731f3c9a-kube-api-access-ptzns" pod "5c38bb28-8f0c-4352-bc91-a259731f3c9a" (UID: "5c38bb28-8f0c-4352-bc91-a259731f3c9a"). InnerVolumeSpecName "kube-api-access-ptzns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:24.997633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:24.997570 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptzns\" (UniqueName: \"kubernetes.io/projected/5c38bb28-8f0c-4352-bc91-a259731f3c9a-kube-api-access-ptzns\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:25.052181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.052078 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:25.108896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.108867 4981 generic.go:356] "Generic (PLEG): container finished" podID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" containerID="96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea" exitCode=0 Nov 01 23:03:25.109083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.108910 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" Nov 01 23:03:25.109083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.108952 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" event={"ID":"5c38bb28-8f0c-4352-bc91-a259731f3c9a","Type":"ContainerDied","Data":"96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea"} Nov 01 23:03:25.109083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.109019 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk" event={"ID":"5c38bb28-8f0c-4352-bc91-a259731f3c9a","Type":"ContainerDied","Data":"dfa8f0e23e0e142d888a9cfbdb70da6d99b026c77a091fe0f16fac11f7396c35"} Nov 01 23:03:25.109083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.109048 4981 scope.go:122] "RemoveContainer" containerID="96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea" Nov 01 23:03:25.110956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.110932 4981 generic.go:356] "Generic (PLEG): container finished" podID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" containerID="09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f" exitCode=0 Nov 01 23:03:25.111164 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.110971 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" Nov 01 23:03:25.111273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.110984 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" event={"ID":"2e75445e-7d5c-4ee1-86b7-9c0a6b874572","Type":"ContainerDied","Data":"09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f"} Nov 01 23:03:25.111273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.111264 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9" event={"ID":"2e75445e-7d5c-4ee1-86b7-9c0a6b874572","Type":"ContainerDied","Data":"381c89d8c64dff44863fe9f9771f00da22b816a07dd969b0f9e4eedc98ba9894"} Nov 01 23:03:25.120627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.120567 4981 scope.go:122] "RemoveContainer" containerID="96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea" Nov 01 23:03:25.120920 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:03:25.120900 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea\": container with ID starting with 96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea not found: ID does not exist" containerID="96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea" Nov 01 23:03:25.121001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.120927 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea"} err="failed to get container status \"96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea\": rpc error: code = NotFound desc = could not find container \"96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea\": container with ID starting with 96578a38d553c829b7debda6df338e43b9ac1f0b75172037872bf30a70341bea not found: ID does not exist" Nov 01 23:03:25.121001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.120944 4981 scope.go:122] "RemoveContainer" containerID="09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f" Nov 01 23:03:25.131229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.131210 4981 scope.go:122] "RemoveContainer" containerID="09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f" Nov 01 23:03:25.131476 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:03:25.131456 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f\": container with ID starting with 09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f not found: ID does not exist" containerID="09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f" Nov 01 23:03:25.131525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.131482 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f"} err="failed to get container status \"09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f\": rpc error: code = NotFound desc = could not find container \"09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f\": container with ID starting with 09a12ad337acb8dbd7d1a300493bed75ce9cad9e0db7374f920cffc1776a942f not found: ID does not exist" Nov 01 23:03:25.139231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.139204 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9"] Nov 01 23:03:25.144087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.144067 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c49240-688d89fd69-km2k9"] Nov 01 23:03:25.155209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.155189 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk"] Nov 01 23:03:25.161588 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.161560 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c49240-6fb5c6d8bc-mbdwk"] Nov 01 23:03:25.507867 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.507834 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:25.608885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.608837 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:25.683068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.682984 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr"] Nov 01 23:03:25.683726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683707 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" containerName="payload-pre-processing" Nov 01 23:03:25.683726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683723 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" containerName="payload-pre-processing" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683748 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a184b51d-c4c0-449d-bd87-dcf50c02c65a" containerName="payload-processing" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683754 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a184b51d-c4c0-449d-bd87-dcf50c02c65a" containerName="payload-processing" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683783 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f0bfac70-72e1-450c-9b18-3137a73cedef" containerName="revoke-keys" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683788 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bfac70-72e1-450c-9b18-3137a73cedef" containerName="revoke-keys" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683804 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" containerName="payload-processing" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683808 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" containerName="payload-processing" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683903 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a184b51d-c4c0-449d-bd87-dcf50c02c65a" containerName="payload-processing" Nov 01 23:03:25.683920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683919 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bfac70-72e1-450c-9b18-3137a73cedef" containerName="revoke-keys" Nov 01 23:03:25.684353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683928 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" containerName="payload-processing" Nov 01 23:03:25.684353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.683936 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" containerName="payload-pre-processing" Nov 01 23:03:25.689007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.688986 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:25.695235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.695214 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr"] Nov 01 23:03:25.785603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.773686 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-337dee\"" Nov 01 23:03:25.785603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.773840 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-337dee-dockercfg-pjdzf\"" Nov 01 23:03:25.817856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.817803 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcc48c4-daf7-4243-9dc4-cc95ed523490-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:25.827901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.827857 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72ln\" (UniqueName: \"kubernetes.io/projected/fdcc48c4-daf7-4243-9dc4-cc95ed523490-kube-api-access-m72ln\") pod \"payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:25.929130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.929095 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72ln\" (UniqueName: \"kubernetes.io/projected/fdcc48c4-daf7-4243-9dc4-cc95ed523490-kube-api-access-m72ln\") pod \"payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:25.929336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.929215 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcc48c4-daf7-4243-9dc4-cc95ed523490-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:25.930013 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.929988 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcc48c4-daf7-4243-9dc4-cc95ed523490-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:25.936883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:25.936827 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72ln\" (UniqueName: \"kubernetes.io/projected/fdcc48c4-daf7-4243-9dc4-cc95ed523490-kube-api-access-m72ln\") pod \"payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:26.006812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:26.006784 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:26.149324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:26.149296 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:26.289136 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:26.289107 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr"] Nov 01 23:03:26.289291 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:03:26.289127 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcc48c4_daf7_4243_9dc4_cc95ed523490.slice/crio-a250f0c38cb93333fd6b71d48a1a6b178fe3b26694b2b5851d93f3c6aafed610 WatchSource:0}: Error finding container a250f0c38cb93333fd6b71d48a1a6b178fe3b26694b2b5851d93f3c6aafed610: Status 404 returned error can't find the container with id a250f0c38cb93333fd6b71d48a1a6b178fe3b26694b2b5851d93f3c6aafed610 Nov 01 23:03:26.291162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:26.291146 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:03:26.737724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:26.737684 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e75445e-7d5c-4ee1-86b7-9c0a6b874572" path="/var/lib/kubelet/pods/2e75445e-7d5c-4ee1-86b7-9c0a6b874572/volumes" Nov 01 23:03:26.738302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:26.738280 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c38bb28-8f0c-4352-bc91-a259731f3c9a" path="/var/lib/kubelet/pods/5c38bb28-8f0c-4352-bc91-a259731f3c9a/volumes" Nov 01 23:03:26.771707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:26.771679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:27.138231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:27.138144 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" event={"ID":"fdcc48c4-daf7-4243-9dc4-cc95ed523490","Type":"ContainerStarted","Data":"a250f0c38cb93333fd6b71d48a1a6b178fe3b26694b2b5851d93f3c6aafed610"} Nov 01 23:03:27.477827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:27.477782 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:28.148233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:28.148143 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" event={"ID":"fdcc48c4-daf7-4243-9dc4-cc95ed523490","Type":"ContainerStarted","Data":"35c241da39ac7f38feac1dfa83fa8bda9352b24ebe20ca7777318edf72c1a28b"} Nov 01 23:03:28.148233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:28.148204 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:28.177534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:28.177334 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" podStartSLOduration=1.705074805 podStartE2EDuration="3.177317156s" totalImagesPullingTime="1.472242351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:28.173542326 +0000 UTC m=+2373.601420259" watchObservedRunningTime="2026-11-01 23:03:28.177317156 +0000 UTC m=+2373.605195088" Nov 01 23:03:28.186145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:28.186110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:28.956725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:28.956689 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:29.583628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:29.583599 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:30.382035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:30.382005 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:31.182153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:31.182114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:31.772207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:31.772151 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:32.374863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:32.374827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:33.079482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:33.079448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:33.769340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:33.769311 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:34.290809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.290773 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4"] Nov 01 23:03:34.291049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.291024 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" podUID="677119b8-8fdb-4607-92ff-1158dda37e4a" containerName="maas-api" containerID="cri-o://fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d" gracePeriod=30 Nov 01 23:03:34.351471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.351436 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:34.495886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.495839 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:34.514886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.514855 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45"] Nov 01 23:03:34.524530 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.524501 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-64b196-zfp45"] Nov 01 23:03:34.582432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.582411 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 23:03:34.588557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.588536 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-maas-api-tls\") pod \"677119b8-8fdb-4607-92ff-1158dda37e4a\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " Nov 01 23:03:34.588721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.588707 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677119b8-8fdb-4607-92ff-1158dda37e4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677119b8-8fdb-4607-92ff-1158dda37e4a-openshift-service-ca\") pod \"677119b8-8fdb-4607-92ff-1158dda37e4a\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " Nov 01 23:03:34.588766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.588751 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677119b8-8fdb-4607-92ff-1158dda37e4a-kube-api-access-4sxdr\" (UniqueName: \"kubernetes.io/projected/677119b8-8fdb-4607-92ff-1158dda37e4a-kube-api-access-4sxdr\") pod \"677119b8-8fdb-4607-92ff-1158dda37e4a\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " Nov 01 23:03:34.588808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.588783 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-metrics-certs\") pod \"677119b8-8fdb-4607-92ff-1158dda37e4a\" (UID: \"677119b8-8fdb-4607-92ff-1158dda37e4a\") " Nov 01 23:03:34.589099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.589074 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677119b8-8fdb-4607-92ff-1158dda37e4a-openshift-service-ca" pod "677119b8-8fdb-4607-92ff-1158dda37e4a" (UID: "677119b8-8fdb-4607-92ff-1158dda37e4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:34.589487 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.589464 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/677119b8-8fdb-4607-92ff-1158dda37e4a-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:34.590733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.590707 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-maas-api-tls" pod "677119b8-8fdb-4607-92ff-1158dda37e4a" (UID: "677119b8-8fdb-4607-92ff-1158dda37e4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:03:34.590943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.590913 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-metrics-certs" pod "677119b8-8fdb-4607-92ff-1158dda37e4a" (UID: "677119b8-8fdb-4607-92ff-1158dda37e4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:03:34.591017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.590914 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677119b8-8fdb-4607-92ff-1158dda37e4a-kube-api-access-4sxdr" pod "677119b8-8fdb-4607-92ff-1158dda37e4a" (UID: "677119b8-8fdb-4607-92ff-1158dda37e4a"). InnerVolumeSpecName "kube-api-access-4sxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:34.595748 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.595728 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:34.772888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.772848 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sxdr\" (UniqueName: \"kubernetes.io/projected/677119b8-8fdb-4607-92ff-1158dda37e4a-kube-api-access-4sxdr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:34.772888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.772884 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:34.773458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.772956 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/677119b8-8fdb-4607-92ff-1158dda37e4a-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:34.821272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.821180 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bfac70-72e1-450c-9b18-3137a73cedef" path="/var/lib/kubelet/pods/f0bfac70-72e1-450c-9b18-3137a73cedef/volumes" Nov 01 23:03:34.971009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:34.970978 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:35.211194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.211157 4981 generic.go:356] "Generic (PLEG): container finished" podID="677119b8-8fdb-4607-92ff-1158dda37e4a" containerID="fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d" exitCode=0 Nov 01 23:03:35.211390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.211232 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" Nov 01 23:03:35.211390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.211243 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" event={"ID":"677119b8-8fdb-4607-92ff-1158dda37e4a","Type":"ContainerDied","Data":"fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d"} Nov 01 23:03:35.211390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.211317 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4" event={"ID":"677119b8-8fdb-4607-92ff-1158dda37e4a","Type":"ContainerDied","Data":"9749b35975879ea40086c5bdefc0c6ea7b7171d47ce920cb1f0bd7230e8943e4"} Nov 01 23:03:35.211390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.211342 4981 scope.go:122] "RemoveContainer" containerID="fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d" Nov 01 23:03:35.222558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.222541 4981 scope.go:122] "RemoveContainer" containerID="fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d" Nov 01 23:03:35.222889 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:03:35.222869 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d\": container with ID starting with fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d not found: ID does not exist" containerID="fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d" Nov 01 23:03:35.222958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.222899 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d"} err="failed to get container status \"fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d\": rpc error: code = NotFound desc = could not find container \"fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d\": container with ID starting with fad1d1af5b6d1c32e5db7bd85aeed8a241c05cf55fe4e38890fad520e9f4a36d not found: ID does not exist" Nov 01 23:03:35.235334 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.235308 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4"] Nov 01 23:03:35.239709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.239685 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64b196-6b4698b899-lstl4"] Nov 01 23:03:35.464054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:35.463961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:36.081181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:36.081132 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:36.738400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:36.738366 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677119b8-8fdb-4607-92ff-1158dda37e4a" path="/var/lib/kubelet/pods/677119b8-8fdb-4607-92ff-1158dda37e4a/volumes" Nov 01 23:03:36.761777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:36.761744 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:37.456517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:37.456490 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:38.139633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:38.139598 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:38.741322 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:38.741296 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:39.158473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:39.158400 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:03:39.573453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:39.573425 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:40.279978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:40.279951 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:41.128314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:41.128283 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:41.231605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:41.224798 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:41.439106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:41.439077 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:41.941517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:41.941484 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:42.268786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.268659 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:42.562012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.561907 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:42.668769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.668735 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7"] Nov 01 23:03:42.669941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.669914 4981 cpu_manager.go:414] "containerMap: removing container" podUID="677119b8-8fdb-4607-92ff-1158dda37e4a" containerName="maas-api" Nov 01 23:03:42.669941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.669942 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677119b8-8fdb-4607-92ff-1158dda37e4a" containerName="maas-api" Nov 01 23:03:42.670192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.670177 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="677119b8-8fdb-4607-92ff-1158dda37e4a" containerName="maas-api" Nov 01 23:03:42.676245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.676222 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.686077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.685619 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3fb5b4-openshift-default-dockercfg-p987m\"" Nov 01 23:03:42.687793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.687762 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7"] Nov 01 23:03:42.701415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.701386 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-certs\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.701638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.701474 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-envoy\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.701919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.701893 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-socket\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.702137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.702114 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-podinfo\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.702289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.702262 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cbedbb-ef98-4fc8-927b-a47634d5e428-istiod-ca-cert\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.702521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.702419 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-token\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.702782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.702535 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-data\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.702782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.702624 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qztt9\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-kube-api-access-qztt9\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.702782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.702754 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-credential-socket\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.720933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.720895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.804723 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-token\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.804845 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-data\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.804887 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qztt9\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-kube-api-access-qztt9\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.804977 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-credential-socket\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.805086 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-certs\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.805163 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-envoy\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.805219 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-socket\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.805269 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-podinfo\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.805300 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cbedbb-ef98-4fc8-927b-a47634d5e428-istiod-ca-cert\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.806197 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cbedbb-ef98-4fc8-927b-a47634d5e428-istiod-ca-cert\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.806935 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-certs\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.807188 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-data\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.807702 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-credential-socket\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.808599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.807949 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-socket\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.880705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.874823 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-podinfo\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.880705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.875481 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qztt9\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-kube-api-access-qztt9\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.880705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.878098 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-token\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.880705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.878841 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-envoy\") pod \"e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:42.889630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.886221 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:42.993862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:42.993825 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:43.291365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:43.291336 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7"] Nov 01 23:03:43.291484 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:03:43.291397 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cbedbb_ef98_4fc8_927b_a47634d5e428.slice/crio-c3bf3876280f2a29ef1aa1c2bff11c9b34b70d92938f3026aadfb195a99fbd8c WatchSource:0}: Error finding container c3bf3876280f2a29ef1aa1c2bff11c9b34b70d92938f3026aadfb195a99fbd8c: Status 404 returned error can't find the container with id c3bf3876280f2a29ef1aa1c2bff11c9b34b70d92938f3026aadfb195a99fbd8c Nov 01 23:03:43.294103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:43.294076 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:03:43.294186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:43.294134 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:03:43.294186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:43.294163 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:03:43.393039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:43.392984 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:44.134407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:44.134376 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:44.235932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:44.235898 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:44.356300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:44.356251 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" event={"ID":"16cbedbb-ef98-4fc8-927b-a47634d5e428","Type":"ContainerStarted","Data":"c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492"} Nov 01 23:03:44.357705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:44.357674 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" event={"ID":"16cbedbb-ef98-4fc8-927b-a47634d5e428","Type":"ContainerStarted","Data":"c3bf3876280f2a29ef1aa1c2bff11c9b34b70d92938f3026aadfb195a99fbd8c"} Nov 01 23:03:44.396546 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:44.396425 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podStartSLOduration=2.396406776 podStartE2EDuration="2.396406776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:44.389562114 +0000 UTC m=+2389.817440055" watchObservedRunningTime="2026-11-01 23:03:44.396406776 +0000 UTC m=+2389.824284708" Nov 01 23:03:44.529880 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:03:44.529804 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:03:44.536299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:44.536267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:44.910494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:44.910464 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:45.001252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:45.001215 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:45.076422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:45.076390 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:45.364731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:45.364697 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:45.365698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:45.365680 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:03:45.479264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:45.479224 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:46.382961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:46.382927 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:46.754132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:46.754101 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:46.872240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:46.872207 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:47.602742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:47.602702 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:48.015560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.015525 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:48.059939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.059909 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq"] Nov 01 23:03:48.064858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.064841 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.070275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.070254 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fb5b4\"" Nov 01 23:03:48.071703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.071678 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq"] Nov 01 23:03:48.073434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.073416 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fb5b4-dockercfg-mrmwl\"" Nov 01 23:03:48.102830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.102797 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz"] Nov 01 23:03:48.107529 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.107509 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.121495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.121466 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz"] Nov 01 23:03:48.125402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.125377 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:48.259656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.237995 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qdh7\" (UniqueName: \"kubernetes.io/projected/3335b691-b745-459e-b7fa-8fba7ac6388d-kube-api-access-9qdh7\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.260290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.260242 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.283776 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.283617 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3335b691-b745-459e-b7fa-8fba7ac6388d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.284171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.284142 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4xr\" (UniqueName: \"kubernetes.io/projected/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-kube-api-access-wg4xr\") pod \"payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.387598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.386619 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.387598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.386690 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3335b691-b745-459e-b7fa-8fba7ac6388d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.387598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.386722 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4xr\" (UniqueName: \"kubernetes.io/projected/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-kube-api-access-wg4xr\") pod \"payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.387598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.386856 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qdh7\" (UniqueName: \"kubernetes.io/projected/3335b691-b745-459e-b7fa-8fba7ac6388d-kube-api-access-9qdh7\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.387892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.387712 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.389921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.388214 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3335b691-b745-459e-b7fa-8fba7ac6388d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.389921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.388812 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:48.405508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.403992 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qdh7\" (UniqueName: \"kubernetes.io/projected/3335b691-b745-459e-b7fa-8fba7ac6388d-kube-api-access-9qdh7\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.405508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.405302 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4xr\" (UniqueName: \"kubernetes.io/projected/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-kube-api-access-wg4xr\") pod \"payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.430505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.430464 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:48.516251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.514854 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:48.687303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.687260 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:48.738887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.738860 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz"] Nov 01 23:03:48.981010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:48.980978 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq"] Nov 01 23:03:48.981676 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:03:48.981649 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3335b691_b745_459e_b7fa_8fba7ac6388d.slice/crio-de2924f3614cf3a66f71347921dd88232f153f3d6a174cd6330b805649b465c4 WatchSource:0}: Error finding container de2924f3614cf3a66f71347921dd88232f153f3d6a174cd6330b805649b465c4: Status 404 returned error can't find the container with id de2924f3614cf3a66f71347921dd88232f153f3d6a174cd6330b805649b465c4 Nov 01 23:03:49.088469 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.088398 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:49.137669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.137567 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh"] Nov 01 23:03:49.147600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.146986 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.153595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.150640 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh"] Nov 01 23:03:49.198272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.198238 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjls4\" (UniqueName: \"kubernetes.io/projected/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-kube-api-access-qjls4\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.198443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.198273 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.198493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.198458 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.229897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.229812 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:03:49.229897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.229856 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:03:49.299836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.299805 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.300139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.300114 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjls4\" (UniqueName: \"kubernetes.io/projected/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-kube-api-access-qjls4\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.300289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.300151 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.300663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.300640 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.302802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.302783 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.308209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.308186 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjls4\" (UniqueName: \"kubernetes.io/projected/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-kube-api-access-qjls4\") pod \"maas-api-revoke-sub-babd16ca-zrjrh\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.404309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.404270 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" event={"ID":"b78a9c5f-7b40-41c3-9e2f-cf581372dc83","Type":"ContainerStarted","Data":"a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047"} Nov 01 23:03:49.404481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.404330 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" event={"ID":"b78a9c5f-7b40-41c3-9e2f-cf581372dc83","Type":"ContainerStarted","Data":"2dd9a0fd893a00b4372e7eafc3086b800b4733568fc7a9c9d978f884ba5a9ac8"} Nov 01 23:03:49.404481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.404362 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:03:49.406000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.405968 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" event={"ID":"3335b691-b745-459e-b7fa-8fba7ac6388d","Type":"ContainerStarted","Data":"44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b"} Nov 01 23:03:49.406118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.406012 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" event={"ID":"3335b691-b745-459e-b7fa-8fba7ac6388d","Type":"ContainerStarted","Data":"de2924f3614cf3a66f71347921dd88232f153f3d6a174cd6330b805649b465c4"} Nov 01 23:03:49.406118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.406033 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:03:49.422915 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.422866 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" podStartSLOduration=1.422853428 podStartE2EDuration="1.422853428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:49.42099944 +0000 UTC m=+2394.848877371" watchObservedRunningTime="2026-11-01 23:03:49.422853428 +0000 UTC m=+2394.850731360" Nov 01 23:03:49.445409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.444201 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" podStartSLOduration=1.444184601 podStartE2EDuration="1.444184601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:49.439326824 +0000 UTC m=+2394.867204749" watchObservedRunningTime="2026-11-01 23:03:49.444184601 +0000 UTC m=+2394.872062532" Nov 01 23:03:49.494113 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.494014 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:49.499665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.499633 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:49.595769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.595737 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:49.619787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.619753 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db"] Nov 01 23:03:49.625787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.625766 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:49.637349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.637186 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db"] Nov 01 23:03:49.643855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.643823 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec0da7\"" Nov 01 23:03:49.696598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.695126 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec0da7-dockercfg-c8b5v\"" Nov 01 23:03:49.735600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.723700 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d9da03-4964-4b04-aaef-300c11c81a62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:49.735600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.723941 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkj9\" (UniqueName: \"kubernetes.io/projected/73d9da03-4964-4b04-aaef-300c11c81a62-kube-api-access-zkkj9\") pod \"payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:49.836755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.834026 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkj9\" (UniqueName: \"kubernetes.io/projected/73d9da03-4964-4b04-aaef-300c11c81a62-kube-api-access-zkkj9\") pod \"payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:49.836755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.834358 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d9da03-4964-4b04-aaef-300c11c81a62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:49.836755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.835324 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d9da03-4964-4b04-aaef-300c11c81a62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:49.843927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.843892 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh"] Nov 01 23:03:49.864716 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.864648 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkj9\" (UniqueName: \"kubernetes.io/projected/73d9da03-4964-4b04-aaef-300c11c81a62-kube-api-access-zkkj9\") pod \"payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:49.970234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:49.970211 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:50.179934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.179907 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:50.272160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.272128 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db"] Nov 01 23:03:50.276320 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:03:50.276287 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d9da03_4964_4b04_aaef_300c11c81a62.slice/crio-59cc89bc59267d39884c4bb456a7d271d3aea39b100e0b3454eeb5e1a117d3fd WatchSource:0}: Error finding container 59cc89bc59267d39884c4bb456a7d271d3aea39b100e0b3454eeb5e1a117d3fd: Status 404 returned error can't find the container with id 59cc89bc59267d39884c4bb456a7d271d3aea39b100e0b3454eeb5e1a117d3fd Nov 01 23:03:50.423438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.423397 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" event={"ID":"73d9da03-4964-4b04-aaef-300c11c81a62","Type":"ContainerStarted","Data":"8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317"} Nov 01 23:03:50.423671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.423506 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" event={"ID":"73d9da03-4964-4b04-aaef-300c11c81a62","Type":"ContainerStarted","Data":"59cc89bc59267d39884c4bb456a7d271d3aea39b100e0b3454eeb5e1a117d3fd"} Nov 01 23:03:50.423803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.423779 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:03:50.425237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.425209 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" event={"ID":"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc","Type":"ContainerStarted","Data":"3a4ccb614fa6f0fda3002e30061a9b2b413922fe9e5652cd3f91902a7a2b5931"} Nov 01 23:03:50.425340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.425258 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" event={"ID":"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc","Type":"ContainerStarted","Data":"f098d7f176a136dabac561d0319b0e263069c0f21406af82dc1d22512c2cb697"} Nov 01 23:03:50.444282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.444178 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" podStartSLOduration=1.444166324 podStartE2EDuration="1.444166324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:50.439095252 +0000 UTC m=+2395.866973184" watchObservedRunningTime="2026-11-01 23:03:50.444166324 +0000 UTC m=+2395.872044251" Nov 01 23:03:50.454718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.454681 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" podStartSLOduration=1.454670427 podStartE2EDuration="1.454670427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:50.453932161 +0000 UTC m=+2395.881810091" watchObservedRunningTime="2026-11-01 23:03:50.454670427 +0000 UTC m=+2395.882548359" Nov 01 23:03:50.899701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:50.899666 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:51.434713 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:51.434682 4981 generic.go:356] "Generic (PLEG): container finished" podID="fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" containerID="3a4ccb614fa6f0fda3002e30061a9b2b413922fe9e5652cd3f91902a7a2b5931" exitCode=0 Nov 01 23:03:51.434884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:51.434766 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" event={"ID":"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc","Type":"ContainerDied","Data":"3a4ccb614fa6f0fda3002e30061a9b2b413922fe9e5652cd3f91902a7a2b5931"} Nov 01 23:03:51.685179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:51.685093 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:52.392624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.392561 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:52.647110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.647050 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:52.690275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.690243 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-kube-api-access-qjls4\" (UniqueName: \"kubernetes.io/projected/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-kube-api-access-qjls4\") pod \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " Nov 01 23:03:52.690538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.690520 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-xks-ca\") pod \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " Nov 01 23:03:52.690638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.690619 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-openshift-ca\") pod \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\" (UID: \"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc\") " Nov 01 23:03:52.691061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.691032 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-openshift-ca" pod "fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" (UID: "fdfacbf6-c481-4776-b4d5-ba300c2ad1cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:03:52.691282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.691263 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:52.693081 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.693047 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-kube-api-access-qjls4" pod "fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" (UID: "fdfacbf6-c481-4776-b4d5-ba300c2ad1cc"). InnerVolumeSpecName "kube-api-access-qjls4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:03:52.693185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.693076 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-xks-ca" pod "fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" (UID: "fdfacbf6-c481-4776-b4d5-ba300c2ad1cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:03:52.792804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.792781 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:52.792804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:52.792803 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjls4\" (UniqueName: \"kubernetes.io/projected/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc-kube-api-access-qjls4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:03:53.143894 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:53.143863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:53.476792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:53.476760 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" Nov 01 23:03:53.476792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:53.476772 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh" event={"ID":"fdfacbf6-c481-4776-b4d5-ba300c2ad1cc","Type":"ContainerDied","Data":"f098d7f176a136dabac561d0319b0e263069c0f21406af82dc1d22512c2cb697"} Nov 01 23:03:53.477268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:53.476860 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f098d7f176a136dabac561d0319b0e263069c0f21406af82dc1d22512c2cb697" Nov 01 23:03:53.852335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:53.852253 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:54.467622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:54.467570 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:55.229163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:55.229120 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:56.034133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:56.034088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:56.727285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:56.727245 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:57.386967 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:57.386940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:58.170041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:58.170009 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:59.281975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.281929 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:03:59.451639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.451608 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb"] Nov 01 23:03:59.452366 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.452351 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" containerName="revoke-keys" Nov 01 23:03:59.452412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.452369 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" containerName="revoke-keys" Nov 01 23:03:59.452516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.452507 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" containerName="revoke-keys" Nov 01 23:03:59.458500 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.458480 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.462619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.462560 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:03:59.462904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.462281 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:03:59.465557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.465534 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb"] Nov 01 23:03:59.517843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.517815 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.518006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.517948 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27qwv\" (UniqueName: \"kubernetes.io/projected/59177733-0923-4dfb-bec2-6004553b3c4e-kube-api-access-27qwv\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.518106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.518082 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.619389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.619311 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.619547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.619389 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27qwv\" (UniqueName: \"kubernetes.io/projected/59177733-0923-4dfb-bec2-6004553b3c4e-kube-api-access-27qwv\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.619547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.619466 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.620170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.620150 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.621708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.621684 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.628026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.628003 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27qwv\" (UniqueName: \"kubernetes.io/projected/59177733-0923-4dfb-bec2-6004553b3c4e-kube-api-access-27qwv\") pod \"maas-api-revoke-sub-601a67c5-tpkmb\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:03:59.777239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:03:59.777208 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:04:00.068639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.064780 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb"] Nov 01 23:04:00.285820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.285793 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:00.428676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.428649 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:04:00.428854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.428713 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:04:00.539111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.539018 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" event={"ID":"59177733-0923-4dfb-bec2-6004553b3c4e","Type":"ContainerStarted","Data":"73e2aa10cabfdf2a7fa2d7baac8571ace3e195271a8852030cf2b1ba4d164589"} Nov 01 23:04:00.539286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.539145 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" event={"ID":"59177733-0923-4dfb-bec2-6004553b3c4e","Type":"ContainerStarted","Data":"3e65cc9d5a070634df0364c221c5219d6a21bacad9596f39905298a674529dd5"} Nov 01 23:04:00.559461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.559411 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" podStartSLOduration=1.559398019 podStartE2EDuration="1.559398019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:00.557824316 +0000 UTC m=+2405.985702245" watchObservedRunningTime="2026-11-01 23:04:00.559398019 +0000 UTC m=+2405.987275991" Nov 01 23:04:00.960287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:00.960253 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:01.438647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:01.438621 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:04:01.548481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:01.548450 4981 generic.go:356] "Generic (PLEG): container finished" podID="59177733-0923-4dfb-bec2-6004553b3c4e" containerID="73e2aa10cabfdf2a7fa2d7baac8571ace3e195271a8852030cf2b1ba4d164589" exitCode=0 Nov 01 23:04:01.548693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:01.548538 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" event={"ID":"59177733-0923-4dfb-bec2-6004553b3c4e","Type":"ContainerDied","Data":"73e2aa10cabfdf2a7fa2d7baac8571ace3e195271a8852030cf2b1ba4d164589"} Nov 01 23:04:01.847361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:01.847274 4981 scope.go:122] "RemoveContainer" containerID="632841cef7982f776c32e5565a9ea708aec606e13f42d21e4c74468304c85fcb" Nov 01 23:04:01.858151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:01.858128 4981 scope.go:122] "RemoveContainer" containerID="137a521bc11c85618f451b47966be3a534c2cc5fb2b797d8494ca8302ef3a108" Nov 01 23:04:01.907798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:01.907769 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:02.617377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.617334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:02.769443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.769415 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:04:02.917068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.916986 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-openshift-ca\") pod \"59177733-0923-4dfb-bec2-6004553b3c4e\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " Nov 01 23:04:02.917236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.917100 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-xks-ca\") pod \"59177733-0923-4dfb-bec2-6004553b3c4e\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " Nov 01 23:04:02.917236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.917161 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59177733-0923-4dfb-bec2-6004553b3c4e-kube-api-access-27qwv\" (UniqueName: \"kubernetes.io/projected/59177733-0923-4dfb-bec2-6004553b3c4e-kube-api-access-27qwv\") pod \"59177733-0923-4dfb-bec2-6004553b3c4e\" (UID: \"59177733-0923-4dfb-bec2-6004553b3c4e\") " Nov 01 23:04:02.917396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.917370 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-openshift-ca" pod "59177733-0923-4dfb-bec2-6004553b3c4e" (UID: "59177733-0923-4dfb-bec2-6004553b3c4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:02.917785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.917761 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:02.919116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.919096 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-xks-ca" pod "59177733-0923-4dfb-bec2-6004553b3c4e" (UID: "59177733-0923-4dfb-bec2-6004553b3c4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:04:02.919313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:02.919286 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59177733-0923-4dfb-bec2-6004553b3c4e-kube-api-access-27qwv" pod "59177733-0923-4dfb-bec2-6004553b3c4e" (UID: "59177733-0923-4dfb-bec2-6004553b3c4e"). InnerVolumeSpecName "kube-api-access-27qwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:03.018784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:03.018749 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59177733-0923-4dfb-bec2-6004553b3c4e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:03.018784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:03.018778 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27qwv\" (UniqueName: \"kubernetes.io/projected/59177733-0923-4dfb-bec2-6004553b3c4e-kube-api-access-27qwv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:03.280620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:03.280587 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:03.566627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:03.566517 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" Nov 01 23:04:03.566627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:03.566512 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb" event={"ID":"59177733-0923-4dfb-bec2-6004553b3c4e","Type":"ContainerDied","Data":"3e65cc9d5a070634df0364c221c5219d6a21bacad9596f39905298a674529dd5"} Nov 01 23:04:03.566825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:03.566657 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e65cc9d5a070634df0364c221c5219d6a21bacad9596f39905298a674529dd5" Nov 01 23:04:04.057781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:04.057743 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:04.156167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:04.156128 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:04.941861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:04.941824 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:05.633556 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:05.633521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:05.737212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:05.737183 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:05.834425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:05.834391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:06.403300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:06.403268 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:07.062553 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:07.062522 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:07.758634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:07.758571 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:08.369644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:08.369616 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:08.983422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:08.983391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:09.770193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.770159 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:09.778912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.778889 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz"] Nov 01 23:04:09.779644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.779630 4981 cpu_manager.go:414] "containerMap: removing container" podUID="59177733-0923-4dfb-bec2-6004553b3c4e" containerName="revoke-keys" Nov 01 23:04:09.779696 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.779645 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59177733-0923-4dfb-bec2-6004553b3c4e" containerName="revoke-keys" Nov 01 23:04:09.779794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.779785 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="59177733-0923-4dfb-bec2-6004553b3c4e" containerName="revoke-keys" Nov 01 23:04:09.783470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.783455 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.791082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.791061 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-53dd05-openshift-default-dockercfg-4ppx6\"" Nov 01 23:04:09.798985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.798963 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz"] Nov 01 23:04:09.870203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870170 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-envoy\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870217 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-data\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870244 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870478 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-certs\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870642 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1eff244-27b3-4175-8e14-b812d92860a9-istiod-ca-cert\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870697 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-istio-token\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870824 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1eff244-27b3-4175-8e14-b812d92860a9-istio-podinfo\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870865 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtt4\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-kube-api-access-tmtt4\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.870909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.870886 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-credential-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972101 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-envoy\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972350 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972163 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-data\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972350 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972198 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972350 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972302 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-certs\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972355 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1eff244-27b3-4175-8e14-b812d92860a9-istiod-ca-cert\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972394 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-istio-token\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972525 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1eff244-27b3-4175-8e14-b812d92860a9-istio-podinfo\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972571 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtt4\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-kube-api-access-tmtt4\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972605 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-data\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972617 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-credential-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972692 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.972749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972739 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-certs\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.973061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.972964 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-credential-socket\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.973154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.973120 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1eff244-27b3-4175-8e14-b812d92860a9-istiod-ca-cert\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.974667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.974646 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-envoy\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.975243 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.975211 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1eff244-27b3-4175-8e14-b812d92860a9-istio-podinfo\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.984750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.984728 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-istio-token\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:09.984906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:09.984889 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtt4\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-kube-api-access-tmtt4\") pod \"e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:10.100838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.100769 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:10.394459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.394429 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz"] Nov 01 23:04:10.395347 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:04:10.395319 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1eff244_27b3_4175_8e14_b812d92860a9.slice/crio-be80fc3d72a8c66ce509c4b8c242d35009b8589e9e4a973b300a8b27c538c67a WatchSource:0}: Error finding container be80fc3d72a8c66ce509c4b8c242d35009b8589e9e4a973b300a8b27c538c67a: Status 404 returned error can't find the container with id be80fc3d72a8c66ce509c4b8c242d35009b8589e9e4a973b300a8b27c538c67a Nov 01 23:04:10.398250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.398220 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:04:10.398359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.398286 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:04:10.398359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.398313 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:04:10.663455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.663372 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" event={"ID":"a1eff244-27b3-4175-8e14-b812d92860a9","Type":"ContainerStarted","Data":"5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00"} Nov 01 23:04:10.663455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.663436 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" event={"ID":"a1eff244-27b3-4175-8e14-b812d92860a9","Type":"ContainerStarted","Data":"be80fc3d72a8c66ce509c4b8c242d35009b8589e9e4a973b300a8b27c538c67a"} Nov 01 23:04:10.719593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.719527 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podStartSLOduration=1.719512894 podStartE2EDuration="1.719512894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:10.716550826 +0000 UTC m=+2416.144428791" watchObservedRunningTime="2026-11-01 23:04:10.719512894 +0000 UTC m=+2416.147390823" Nov 01 23:04:10.746599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.745948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:10.836960 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:10.836931 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:11.101656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:11.101614 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:11.107072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:11.107036 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 01 23:04:11.107250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:11.107100 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 01 23:04:11.169532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:11.167170 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:11.677601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:11.672822 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:12.101846 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:12.101755 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 01 23:04:12.101846 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:12.101817 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 01 23:04:12.270035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:12.270002 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:13.054592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:13.054546 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:13.101288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:13.101252 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 01 23:04:13.101471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:13.101314 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 01 23:04:13.869951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:13.869922 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:14.101826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:14.101776 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 01 23:04:14.101991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:14.101866 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 01 23:04:14.526624 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:04:14.526541 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:04:14.622944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:14.622912 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:14.968487 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:14.968452 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:15.101956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:15.101914 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 01 23:04:15.102138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:15.101980 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 01 23:04:15.398924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:15.398835 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:15.548140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:15.548106 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:16.102200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:16.102159 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Nov 01 23:04:16.102649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:16.102236 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Nov 01 23:04:16.245827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:16.245779 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:17.106073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:17.106042 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:17.106558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:17.106326 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:17.106828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:17.106813 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:17.107506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:17.107486 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:04:17.912231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:17.912196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:18.020497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:18.020461 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:18.598103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:18.598063 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:18.915397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:18.915322 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:19.038090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:19.038051 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:19.150186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:19.150150 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:19.862162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:19.862121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:20.857415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:20.857380 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:21.583056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:21.583026 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:22.569960 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:22.569930 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:23.161387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:23.161350 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:23.291947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:23.291918 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:23.812504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:23.812474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:24.372592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:24.372548 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:24.493494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:24.493457 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:25.282319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:25.282276 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:26.419815 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:26.419783 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:27.123993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:27.123959 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz"] Nov 01 23:04:27.125298 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:27.125266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:27.136365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:27.136335 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq"] Nov 01 23:04:27.224599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:27.220644 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:27.438107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:27.438063 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:27.929839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:27.929806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:28.477273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:28.477229 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db"] Nov 01 23:04:28.551033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:28.551000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:28.667012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:28.666982 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:28.818865 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:28.818791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:28.974493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:28.974456 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:29.595278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:29.595248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:30.301136 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:30.301102 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:31.163812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:31.163776 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:31.856051 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:31.856012 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:32.125179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.125087 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" podUID="b78a9c5f-7b40-41c3-9e2f-cf581372dc83" containerName="payload-processing" containerID="cri-o://a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047" gracePeriod=25 Nov 01 23:04:32.137642 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.137590 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" podUID="3335b691-b745-459e-b7fa-8fba7ac6388d" containerName="payload-pre-processing" containerID="cri-o://44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b" gracePeriod=25 Nov 01 23:04:32.439809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.439779 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:04:32.505886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.505849 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:04:32.519505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.519471 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3335b691-b745-459e-b7fa-8fba7ac6388d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3335b691-b745-459e-b7fa-8fba7ac6388d-plugins-config-volume\") pod \"3335b691-b745-459e-b7fa-8fba7ac6388d\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " Nov 01 23:04:32.519718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.519672 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-plugins-config-volume\") pod \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " Nov 01 23:04:32.519788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.519745 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-kube-api-access-wg4xr\" (UniqueName: \"kubernetes.io/projected/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-kube-api-access-wg4xr\") pod \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\" (UID: \"b78a9c5f-7b40-41c3-9e2f-cf581372dc83\") " Nov 01 23:04:32.519853 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.519807 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3335b691-b745-459e-b7fa-8fba7ac6388d-kube-api-access-9qdh7\" (UniqueName: \"kubernetes.io/projected/3335b691-b745-459e-b7fa-8fba7ac6388d-kube-api-access-9qdh7\") pod \"3335b691-b745-459e-b7fa-8fba7ac6388d\" (UID: \"3335b691-b745-459e-b7fa-8fba7ac6388d\") " Nov 01 23:04:32.519915 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.519889 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3335b691-b745-459e-b7fa-8fba7ac6388d-plugins-config-volume" pod "3335b691-b745-459e-b7fa-8fba7ac6388d" (UID: "3335b691-b745-459e-b7fa-8fba7ac6388d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:32.520133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.520109 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-plugins-config-volume" pod "b78a9c5f-7b40-41c3-9e2f-cf581372dc83" (UID: "b78a9c5f-7b40-41c3-9e2f-cf581372dc83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:32.520486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.520465 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3335b691-b745-459e-b7fa-8fba7ac6388d-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:32.520486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.520487 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:32.522074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.522042 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3335b691-b745-459e-b7fa-8fba7ac6388d-kube-api-access-9qdh7" pod "3335b691-b745-459e-b7fa-8fba7ac6388d" (UID: "3335b691-b745-459e-b7fa-8fba7ac6388d"). InnerVolumeSpecName "kube-api-access-9qdh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:32.522074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.522060 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-kube-api-access-wg4xr" pod "b78a9c5f-7b40-41c3-9e2f-cf581372dc83" (UID: "b78a9c5f-7b40-41c3-9e2f-cf581372dc83"). InnerVolumeSpecName "kube-api-access-wg4xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:32.579660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.579622 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:32.635864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.635826 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg4xr\" (UniqueName: \"kubernetes.io/projected/b78a9c5f-7b40-41c3-9e2f-cf581372dc83-kube-api-access-wg4xr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:32.635864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.635864 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qdh7\" (UniqueName: \"kubernetes.io/projected/3335b691-b745-459e-b7fa-8fba7ac6388d-kube-api-access-9qdh7\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:32.881945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.881903 4981 generic.go:356] "Generic (PLEG): container finished" podID="b78a9c5f-7b40-41c3-9e2f-cf581372dc83" containerID="a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047" exitCode=0 Nov 01 23:04:32.881945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.881938 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" event={"ID":"b78a9c5f-7b40-41c3-9e2f-cf581372dc83","Type":"ContainerDied","Data":"a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047"} Nov 01 23:04:32.882208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.881953 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" Nov 01 23:04:32.882208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.881999 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz" event={"ID":"b78a9c5f-7b40-41c3-9e2f-cf581372dc83","Type":"ContainerDied","Data":"2dd9a0fd893a00b4372e7eafc3086b800b4733568fc7a9c9d978f884ba5a9ac8"} Nov 01 23:04:32.882208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.882029 4981 scope.go:122] "RemoveContainer" containerID="a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047" Nov 01 23:04:32.883655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.883628 4981 generic.go:356] "Generic (PLEG): container finished" podID="3335b691-b745-459e-b7fa-8fba7ac6388d" containerID="44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b" exitCode=0 Nov 01 23:04:32.883759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.883676 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" Nov 01 23:04:32.883759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.883703 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" event={"ID":"3335b691-b745-459e-b7fa-8fba7ac6388d","Type":"ContainerDied","Data":"44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b"} Nov 01 23:04:32.883759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.883742 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq" event={"ID":"3335b691-b745-459e-b7fa-8fba7ac6388d","Type":"ContainerDied","Data":"de2924f3614cf3a66f71347921dd88232f153f3d6a174cd6330b805649b465c4"} Nov 01 23:04:32.897378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.897350 4981 scope.go:122] "RemoveContainer" containerID="a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047" Nov 01 23:04:32.897778 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:04:32.897750 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047\": container with ID starting with a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047 not found: ID does not exist" containerID="a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047" Nov 01 23:04:32.897902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.897784 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047"} err="failed to get container status \"a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047\": rpc error: code = NotFound desc = could not find container \"a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047\": container with ID starting with a609896814d1bee8e348262cdb035641f95f2771756d00e6f632c6bda352b047 not found: ID does not exist" Nov 01 23:04:32.897902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.897800 4981 scope.go:122] "RemoveContainer" containerID="44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b" Nov 01 23:04:32.909692 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.909560 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq"] Nov 01 23:04:32.912040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.912016 4981 scope.go:122] "RemoveContainer" containerID="44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b" Nov 01 23:04:32.912405 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:04:32.912383 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b\": container with ID starting with 44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b not found: ID does not exist" containerID="44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b" Nov 01 23:04:32.912483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.912415 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b"} err="failed to get container status \"44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b\": rpc error: code = NotFound desc = could not find container \"44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b\": container with ID starting with 44c8e0b24543a3739fff8f76e681c6d2463b5661fa7503995d26055e9d87113b not found: ID does not exist" Nov 01 23:04:32.914217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.914197 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-6f7945b6c5-xpdgq"] Nov 01 23:04:32.928418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.928386 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz"] Nov 01 23:04:32.932534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:32.932512 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-5577fd5cdf-fm4hz"] Nov 01 23:04:33.245829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.245796 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:33.478142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.478100 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" podUID="73d9da03-4964-4b04-aaef-300c11c81a62" containerName="payload-processing" containerID="cri-o://8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317" gracePeriod=25 Nov 01 23:04:33.783891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.783866 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:04:33.861800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.861774 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d9da03-4964-4b04-aaef-300c11c81a62-kube-api-access-zkkj9\" (UniqueName: \"kubernetes.io/projected/73d9da03-4964-4b04-aaef-300c11c81a62-kube-api-access-zkkj9\") pod \"73d9da03-4964-4b04-aaef-300c11c81a62\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " Nov 01 23:04:33.861987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.861941 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d9da03-4964-4b04-aaef-300c11c81a62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d9da03-4964-4b04-aaef-300c11c81a62-plugins-config-volume\") pod \"73d9da03-4964-4b04-aaef-300c11c81a62\" (UID: \"73d9da03-4964-4b04-aaef-300c11c81a62\") " Nov 01 23:04:33.862306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.862280 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d9da03-4964-4b04-aaef-300c11c81a62-plugins-config-volume" pod "73d9da03-4964-4b04-aaef-300c11c81a62" (UID: "73d9da03-4964-4b04-aaef-300c11c81a62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:33.862757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.862736 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d9da03-4964-4b04-aaef-300c11c81a62-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:33.864035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.864013 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d9da03-4964-4b04-aaef-300c11c81a62-kube-api-access-zkkj9" pod "73d9da03-4964-4b04-aaef-300c11c81a62" (UID: "73d9da03-4964-4b04-aaef-300c11c81a62"). InnerVolumeSpecName "kube-api-access-zkkj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:33.892842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.892812 4981 generic.go:356] "Generic (PLEG): container finished" podID="73d9da03-4964-4b04-aaef-300c11c81a62" containerID="8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317" exitCode=0 Nov 01 23:04:33.892994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.892859 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" Nov 01 23:04:33.892994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.892905 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" event={"ID":"73d9da03-4964-4b04-aaef-300c11c81a62","Type":"ContainerDied","Data":"8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317"} Nov 01 23:04:33.892994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.892962 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db" event={"ID":"73d9da03-4964-4b04-aaef-300c11c81a62","Type":"ContainerDied","Data":"59cc89bc59267d39884c4bb456a7d271d3aea39b100e0b3454eeb5e1a117d3fd"} Nov 01 23:04:33.892994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.892988 4981 scope.go:122] "RemoveContainer" containerID="8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317" Nov 01 23:04:33.920815 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.920787 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:33.923184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.923156 4981 scope.go:122] "RemoveContainer" containerID="8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317" Nov 01 23:04:33.923470 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:04:33.923447 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317\": container with ID starting with 8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317 not found: ID does not exist" containerID="8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317" Nov 01 23:04:33.923552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.923482 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317"} err="failed to get container status \"8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317\": rpc error: code = NotFound desc = could not find container \"8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317\": container with ID starting with 8729bd6c788c50f952eba25b998ba22bbe0687c3a8b8f7a54f831044e8917317 not found: ID does not exist" Nov 01 23:04:33.924364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.924350 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db"] Nov 01 23:04:33.928256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.928236 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-765999774c-7j6db"] Nov 01 23:04:33.978647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:33.974913 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkj9\" (UniqueName: \"kubernetes.io/projected/73d9da03-4964-4b04-aaef-300c11c81a62-kube-api-access-zkkj9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:34.678383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:34.678349 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:34.778721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:34.778692 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3335b691-b745-459e-b7fa-8fba7ac6388d" path="/var/lib/kubelet/pods/3335b691-b745-459e-b7fa-8fba7ac6388d/volumes" Nov 01 23:04:34.779189 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:34.779175 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d9da03-4964-4b04-aaef-300c11c81a62" path="/var/lib/kubelet/pods/73d9da03-4964-4b04-aaef-300c11c81a62/volumes" Nov 01 23:04:34.779559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:34.779546 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78a9c5f-7b40-41c3-9e2f-cf581372dc83" path="/var/lib/kubelet/pods/b78a9c5f-7b40-41c3-9e2f-cf581372dc83/volumes" Nov 01 23:04:35.362881 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:35.362850 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:35.987855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:35.987817 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:36.836082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:36.836029 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:37.460187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:37.460126 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:38.060800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:38.060764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:38.672674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:38.672638 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:39.362939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:39.362908 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:39.978193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:39.978138 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:40.547302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.547261 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq"] Nov 01 23:04:40.548474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548444 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3335b691-b745-459e-b7fa-8fba7ac6388d" containerName="payload-pre-processing" Nov 01 23:04:40.548474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548473 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3335b691-b745-459e-b7fa-8fba7ac6388d" containerName="payload-pre-processing" Nov 01 23:04:40.548730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548498 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b78a9c5f-7b40-41c3-9e2f-cf581372dc83" containerName="payload-processing" Nov 01 23:04:40.548730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548506 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78a9c5f-7b40-41c3-9e2f-cf581372dc83" containerName="payload-processing" Nov 01 23:04:40.548730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548527 4981 cpu_manager.go:414] "containerMap: removing container" podUID="73d9da03-4964-4b04-aaef-300c11c81a62" containerName="payload-processing" Nov 01 23:04:40.548730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548536 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d9da03-4964-4b04-aaef-300c11c81a62" containerName="payload-processing" Nov 01 23:04:40.548970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548759 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3335b691-b745-459e-b7fa-8fba7ac6388d" containerName="payload-pre-processing" Nov 01 23:04:40.548970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548777 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78a9c5f-7b40-41c3-9e2f-cf581372dc83" containerName="payload-processing" Nov 01 23:04:40.548970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.548787 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d9da03-4964-4b04-aaef-300c11c81a62" containerName="payload-processing" Nov 01 23:04:40.554584 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.554554 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.561152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.561127 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8e3381-serving-cert\"" Nov 01 23:04:40.567570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.567540 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq"] Nov 01 23:04:40.589991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.589955 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:40.701454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.701395 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.704212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.704179 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.704435 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.704418 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx7t\" (UniqueName: \"kubernetes.io/projected/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-kube-api-access-wtx7t\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.704668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.704648 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.705045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.702204 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:40.806002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.805911 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.806197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.806057 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.806197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.806079 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx7t\" (UniqueName: \"kubernetes.io/projected/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-kube-api-access-wtx7t\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.806197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.806125 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.806866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.806839 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.808446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.808419 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.808536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.808453 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-metrics-certs\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.813725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.813695 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx7t\" (UniqueName: \"kubernetes.io/projected/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-kube-api-access-wtx7t\") pod \"maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:40.841196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.841162 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:40.879102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:40.879067 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:41.166673 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:04:41.166640 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fe4d0e_ec09_4028_a39b_11b2e9b1e72e.slice/crio-30cfe63b9c22c00836e839b415975e63f5481451e0bedd20601e0005dfff9279 WatchSource:0}: Error finding container 30cfe63b9c22c00836e839b415975e63f5481451e0bedd20601e0005dfff9279: Status 404 returned error can't find the container with id 30cfe63b9c22c00836e839b415975e63f5481451e0bedd20601e0005dfff9279 Nov 01 23:04:41.167101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.166740 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq"] Nov 01 23:04:41.452606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.452548 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:41.844195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.844163 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm"] Nov 01 23:04:41.844402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.844379 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" containerID="cri-o://8ada89906faa209141894878630924d9a00dc4465d8ecc2ac3c086da220ce83a" gracePeriod=30 Nov 01 23:04:41.931927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.931877 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:41.980259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.980156 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" event={"ID":"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e","Type":"ContainerStarted","Data":"cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271"} Nov 01 23:04:41.980259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.980259 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:41.980457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:41.980278 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" event={"ID":"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e","Type":"ContainerStarted","Data":"30cfe63b9c22c00836e839b415975e63f5481451e0bedd20601e0005dfff9279"} Nov 01 23:04:42.002028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.001979 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" podStartSLOduration=1.531375956 podStartE2EDuration="2.001966253s" totalImagesPullingTime="470.590297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:41.996888002 +0000 UTC m=+2447.424765978" watchObservedRunningTime="2026-11-01 23:04:42.001966253 +0000 UTC m=+2447.429844195" Nov 01 23:04:42.032810 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.032757 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2"] Nov 01 23:04:42.038764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.038735 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.039622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.039526 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb"] Nov 01 23:04:42.041568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.041538 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:42.046732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.046706 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5a8d8d-99bb-4706-9481-0320e3ff280f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.046874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.046831 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svl22\" (UniqueName: \"kubernetes.io/projected/ff5a8d8d-99bb-4706-9481-0320e3ff280f-kube-api-access-svl22\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.052559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.048760 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fb5b4-dockercfg-zg2zw\"" Nov 01 23:04:42.052559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.048879 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fb5b4\"" Nov 01 23:04:42.052559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.052111 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.056534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.056515 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2"] Nov 01 23:04:42.066304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.063646 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb"] Nov 01 23:04:42.162629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.153297 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5a8d8d-99bb-4706-9481-0320e3ff280f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.169114 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.167817 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svl22\" (UniqueName: \"kubernetes.io/projected/ff5a8d8d-99bb-4706-9481-0320e3ff280f-kube-api-access-svl22\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.169114 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.166623 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5a8d8d-99bb-4706-9481-0320e3ff280f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.224602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.216687 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svl22\" (UniqueName: \"kubernetes.io/projected/ff5a8d8d-99bb-4706-9481-0320e3ff280f-kube-api-access-svl22\") pod \"payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.294714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.294122 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded9f23f-6082-4430-98c9-22b10e78ae0c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.294911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.294713 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7zn\" (UniqueName: \"kubernetes.io/projected/ded9f23f-6082-4430-98c9-22b10e78ae0c-kube-api-access-pv7zn\") pod \"payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.389593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.388908 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:42.425655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.425626 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7zn\" (UniqueName: \"kubernetes.io/projected/ded9f23f-6082-4430-98c9-22b10e78ae0c-kube-api-access-pv7zn\") pod \"payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.425824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.425748 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded9f23f-6082-4430-98c9-22b10e78ae0c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.426446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.426422 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded9f23f-6082-4430-98c9-22b10e78ae0c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.434887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.434865 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7zn\" (UniqueName: \"kubernetes.io/projected/ded9f23f-6082-4430-98c9-22b10e78ae0c-kube-api-access-pv7zn\") pod \"payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.554087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.553999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:42.690226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.690196 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2"] Nov 01 23:04:42.690397 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:04:42.690372 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5a8d8d_99bb_4706_9481_0320e3ff280f.slice/crio-2c168489f88173e803e61f82107f46a0c52fbdfbc3198cffd743f1f8a2ff09a3 WatchSource:0}: Error finding container 2c168489f88173e803e61f82107f46a0c52fbdfbc3198cffd743f1f8a2ff09a3: Status 404 returned error can't find the container with id 2c168489f88173e803e61f82107f46a0c52fbdfbc3198cffd743f1f8a2ff09a3 Nov 01 23:04:42.690482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.690469 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:42.982449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.982416 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb"] Nov 01 23:04:42.983432 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:04:42.983406 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded9f23f_6082_4430_98c9_22b10e78ae0c.slice/crio-559f3c66e4d1569b9d77839ae1184d832a968da5aef41685f73f88eec7f89327 WatchSource:0}: Error finding container 559f3c66e4d1569b9d77839ae1184d832a968da5aef41685f73f88eec7f89327: Status 404 returned error can't find the container with id 559f3c66e4d1569b9d77839ae1184d832a968da5aef41685f73f88eec7f89327 Nov 01 23:04:42.991784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.991755 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" event={"ID":"ff5a8d8d-99bb-4706-9481-0320e3ff280f","Type":"ContainerStarted","Data":"a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957"} Nov 01 23:04:42.991901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.991811 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" event={"ID":"ff5a8d8d-99bb-4706-9481-0320e3ff280f","Type":"ContainerStarted","Data":"2c168489f88173e803e61f82107f46a0c52fbdfbc3198cffd743f1f8a2ff09a3"} Nov 01 23:04:42.992002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:42.991984 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:43.015718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.015672 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" podStartSLOduration=1.015657949 podStartE2EDuration="1.015657949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:43.011452028 +0000 UTC m=+2448.439329961" watchObservedRunningTime="2026-11-01 23:04:43.015657949 +0000 UTC m=+2448.443535879" Nov 01 23:04:43.349616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.349520 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:43.834735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.834700 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:43.929651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.929611 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8"] Nov 01 23:04:43.934814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.934795 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:43.941341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.941318 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec0da7\"" Nov 01 23:04:43.941674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.941652 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec0da7-dockercfg-gwcks\"" Nov 01 23:04:43.945876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.945842 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8"] Nov 01 23:04:43.948368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.948322 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:43.980680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.980653 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw"] Nov 01 23:04:43.986449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:43.986428 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.015637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.011442 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw"] Nov 01 23:04:44.092655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.073342 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" event={"ID":"ded9f23f-6082-4430-98c9-22b10e78ae0c","Type":"ContainerStarted","Data":"48f7dbcef14d41a0ce215e8536c59b51c7a9bf6a3702105790bc7322cacf53a4"} Nov 01 23:04:44.092655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.073434 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" event={"ID":"ded9f23f-6082-4430-98c9-22b10e78ae0c","Type":"ContainerStarted","Data":"559f3c66e4d1569b9d77839ae1184d832a968da5aef41685f73f88eec7f89327"} Nov 01 23:04:44.105417 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.105389 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:44.129673 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.125950 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwr2\" (UniqueName: \"kubernetes.io/projected/6c81b530-835e-4ec1-aacb-b05d5b8efc37-kube-api-access-8rwr2\") pod \"payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:44.129673 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.126007 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgz6h\" (UniqueName: \"kubernetes.io/projected/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-kube-api-access-pgz6h\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.129673 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.126123 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c81b530-835e-4ec1-aacb-b05d5b8efc37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:44.129673 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.126367 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.269647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.241136 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.269647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.241394 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwr2\" (UniqueName: \"kubernetes.io/projected/6c81b530-835e-4ec1-aacb-b05d5b8efc37-kube-api-access-8rwr2\") pod \"payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:44.269647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.241436 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgz6h\" (UniqueName: \"kubernetes.io/projected/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-kube-api-access-pgz6h\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.269647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.241503 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c81b530-835e-4ec1-aacb-b05d5b8efc37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:44.269647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.268179 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.353960 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.353882 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgz6h\" (UniqueName: \"kubernetes.io/projected/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-kube-api-access-pgz6h\") pod \"payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.384338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.384304 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c81b530-835e-4ec1-aacb-b05d5b8efc37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:44.401615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.400007 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwr2\" (UniqueName: \"kubernetes.io/projected/6c81b530-835e-4ec1-aacb-b05d5b8efc37-kube-api-access-8rwr2\") pod \"payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:44.472804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.472563 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:44.567145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.567105 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:44.571538 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:04:44.571497 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:04:44.610269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.610198 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:44.854622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.854548 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" podStartSLOduration=2.854530213 podStartE2EDuration="2.854530213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:44.169774385 +0000 UTC m=+2449.597652317" watchObservedRunningTime="2026-11-01 23:04:44.854530213 +0000 UTC m=+2450.282408148" Nov 01 23:04:44.859219 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:04:44.859174 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c81b530_835e_4ec1_aacb_b05d5b8efc37.slice/crio-6f1af79da688ba65b26348d4b4b88c6cee3054c6545b2dae21ca06798cbd97a3 WatchSource:0}: Error finding container 6f1af79da688ba65b26348d4b4b88c6cee3054c6545b2dae21ca06798cbd97a3: Status 404 returned error can't find the container with id 6f1af79da688ba65b26348d4b4b88c6cee3054c6545b2dae21ca06798cbd97a3 Nov 01 23:04:44.859510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.859489 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8"] Nov 01 23:04:44.929692 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:04:44.929661 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edf3a9a_a7ca_4a51_ba63_eb40609e5357.slice/crio-0d81979d6ca3aba8bafe087f305df6635acd42e393f8a4dea055452142d143e3 WatchSource:0}: Error finding container 0d81979d6ca3aba8bafe087f305df6635acd42e393f8a4dea055452142d143e3: Status 404 returned error can't find the container with id 0d81979d6ca3aba8bafe087f305df6635acd42e393f8a4dea055452142d143e3 Nov 01 23:04:44.931148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:44.931123 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw"] Nov 01 23:04:45.070772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.070734 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" event={"ID":"9edf3a9a-a7ca-4a51-ba63-eb40609e5357","Type":"ContainerStarted","Data":"bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07"} Nov 01 23:04:45.070964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.070785 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" event={"ID":"9edf3a9a-a7ca-4a51-ba63-eb40609e5357","Type":"ContainerStarted","Data":"0d81979d6ca3aba8bafe087f305df6635acd42e393f8a4dea055452142d143e3"} Nov 01 23:04:45.070964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.070830 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:45.072655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.072629 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" event={"ID":"6c81b530-835e-4ec1-aacb-b05d5b8efc37","Type":"ContainerStarted","Data":"92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5"} Nov 01 23:04:45.072818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.072680 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" event={"ID":"6c81b530-835e-4ec1-aacb-b05d5b8efc37","Type":"ContainerStarted","Data":"6f1af79da688ba65b26348d4b4b88c6cee3054c6545b2dae21ca06798cbd97a3"} Nov 01 23:04:45.072818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.072710 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:45.091624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.091564 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" podStartSLOduration=2.091547836 podStartE2EDuration="2.091547836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:45.090920607 +0000 UTC m=+2450.518798540" watchObservedRunningTime="2026-11-01 23:04:45.091547836 +0000 UTC m=+2450.519425771" Nov 01 23:04:45.121007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.120904 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" podStartSLOduration=2.120885797 podStartE2EDuration="2.120885797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:45.10824132 +0000 UTC m=+2450.536119254" watchObservedRunningTime="2026-11-01 23:04:45.120885797 +0000 UTC m=+2450.548763731" Nov 01 23:04:45.203255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:45.203223 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:46.037736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:46.037698 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d"] Nov 01 23:04:46.042111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:46.042083 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb"] Nov 01 23:04:46.050671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:46.050640 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ecfa7c1-6tm9d"] Nov 01 23:04:46.056727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:46.056702 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ffd568-k9rzb"] Nov 01 23:04:46.086831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:46.086796 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:46.814597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:46.801819 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4dbbb9-0a35-44f8-a1af-91e065b368b5" path="/var/lib/kubelet/pods/0c4dbbb9-0a35-44f8-a1af-91e065b368b5/volumes" Nov 01 23:04:46.814597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:46.802558 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dad7027-ccd0-46ad-b7d1-2e7236ec096a" path="/var/lib/kubelet/pods/9dad7027-ccd0-46ad-b7d1-2e7236ec096a/volumes" Nov 01 23:04:47.115651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.097947 4981 generic.go:356] "Generic (PLEG): container finished" podID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerID="8ada89906faa209141894878630924d9a00dc4465d8ecc2ac3c086da220ce83a" exitCode=0 Nov 01 23:04:47.115651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.098255 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" event={"ID":"1e14a2a1-e3bd-4bb4-a477-06d73f87df06","Type":"ContainerDied","Data":"8ada89906faa209141894878630924d9a00dc4465d8ecc2ac3c086da220ce83a"} Nov 01 23:04:47.115651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.098313 4981 scope.go:122] "RemoveContainer" containerID="815bba694e5437fd06c34f709cf97999d5fc90151a8d55762db0ef48f430e587" Nov 01 23:04:47.172733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.172694 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:47.383635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.383558 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548218 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-podinfo\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548495 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-token\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-token\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548546 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-credential-socket\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548643 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istiod-ca-cert\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548762 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-data\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548822 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-socket\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548861 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-envoy\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.548921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548902 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-kube-api-access-s6sd8\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-kube-api-access-s6sd8\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.549550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.548941 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-certs\") pod \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\" (UID: \"1e14a2a1-e3bd-4bb4-a477-06d73f87df06\") " Nov 01 23:04:47.549744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.549691 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istiod-ca-cert" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:47.549974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.549926 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-certs" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:47.550195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.550130 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-data" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:47.550513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.550487 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-socket" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:47.551036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.551010 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-credential-socket" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:47.554442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.554415 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-envoy" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:04:47.554639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.554509 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-podinfo" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:04:47.554639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.554510 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-token" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:47.555482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.555463 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-kube-api-access-s6sd8" pod "1e14a2a1-e3bd-4bb4-a477-06d73f87df06" (UID: "1e14a2a1-e3bd-4bb4-a477-06d73f87df06"). InnerVolumeSpecName "kube-api-access-s6sd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.650928 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.650973 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.650983 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.650992 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.651001 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.651010 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.651018 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6sd8\" (UniqueName: \"kubernetes.io/projected/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-kube-api-access-s6sd8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.651028 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.651037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.651036 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e14a2a1-e3bd-4bb4-a477-06d73f87df06-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:47.995522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.995489 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:47.997667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:47.997646 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:04:48.111699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:48.111666 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" Nov 01 23:04:48.111896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:48.111666 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm" event={"ID":"1e14a2a1-e3bd-4bb4-a477-06d73f87df06","Type":"ContainerDied","Data":"849d1385fa644d395680ea024bae14557343aac5bed7b04df3476fd9141e1935"} Nov 01 23:04:48.111896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:48.111809 4981 scope.go:122] "RemoveContainer" containerID="8ada89906faa209141894878630924d9a00dc4465d8ecc2ac3c086da220ce83a" Nov 01 23:04:48.160039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:48.159992 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm"] Nov 01 23:04:48.176908 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:48.176874 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2f3705-openshift-default-574ddf7987-pzbnm"] Nov 01 23:04:48.738073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:48.738041 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" path="/var/lib/kubelet/pods/1e14a2a1-e3bd-4bb4-a477-06d73f87df06/volumes" Nov 01 23:04:48.865736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:48.865706 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:49.490137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.490102 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc"] Nov 01 23:04:49.490900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.490884 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" Nov 01 23:04:49.490900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.490900 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" Nov 01 23:04:49.491032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.490915 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" Nov 01 23:04:49.491032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.490921 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" Nov 01 23:04:49.491103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.491063 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" Nov 01 23:04:49.491103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.491074 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e14a2a1-e3bd-4bb4-a477-06d73f87df06" containerName="istio-proxy" Nov 01 23:04:49.496963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.496937 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.511280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.502049 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:04:49.511280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.503200 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:04:49.511280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.504031 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc"] Nov 01 23:04:49.658751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.658719 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:49.696327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.696280 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.696537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.696515 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.696595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.696556 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmr96\" (UniqueName: \"kubernetes.io/projected/2a41e36f-ca5e-4e8a-b322-c801592f61a4-kube-api-access-vmr96\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.797878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.797775 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.797878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.797837 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmr96\" (UniqueName: \"kubernetes.io/projected/2a41e36f-ca5e-4e8a-b322-c801592f61a4-kube-api-access-vmr96\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.798124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.797966 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.798686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.798667 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.800284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.800264 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.805340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.805319 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmr96\" (UniqueName: \"kubernetes.io/projected/2a41e36f-ca5e-4e8a-b322-c801592f61a4-kube-api-access-vmr96\") pod \"maas-api-revoke-sub-fe0d3bbd-vpkcc\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:49.826892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:49.826862 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:50.113449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:50.113422 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc"] Nov 01 23:04:50.116119 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:04:50.116084 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a41e36f_ca5e_4e8a_b322_c801592f61a4.slice/crio-5352782aa03739d094cc60e8250f4275e51be4a85fdbb0e8c7b1c86c28084449 WatchSource:0}: Error finding container 5352782aa03739d094cc60e8250f4275e51be4a85fdbb0e8c7b1c86c28084449: Status 404 returned error can't find the container with id 5352782aa03739d094cc60e8250f4275e51be4a85fdbb0e8c7b1c86c28084449 Nov 01 23:04:50.132782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:50.132753 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" event={"ID":"2a41e36f-ca5e-4e8a-b322-c801592f61a4","Type":"ContainerStarted","Data":"5352782aa03739d094cc60e8250f4275e51be4a85fdbb0e8c7b1c86c28084449"} Nov 01 23:04:50.285639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:50.285571 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:51.001678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:50.996355 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:51.142318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:51.142282 4981 generic.go:356] "Generic (PLEG): container finished" podID="2a41e36f-ca5e-4e8a-b322-c801592f61a4" containerID="c6c0e93528724c5b17af7633fd449477e9e88d88f2bfd21890d3e1dd1ae13eff" exitCode=0 Nov 01 23:04:51.142474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:51.142366 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" event={"ID":"2a41e36f-ca5e-4e8a-b322-c801592f61a4","Type":"ContainerDied","Data":"c6c0e93528724c5b17af7633fd449477e9e88d88f2bfd21890d3e1dd1ae13eff"} Nov 01 23:04:51.762885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:51.762851 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:52.348953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.348924 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:52.373816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.373782 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:52.453551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.453510 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-openshift-ca\") pod \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " Nov 01 23:04:52.455522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.455494 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-xks-ca\") pod \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " Nov 01 23:04:52.456229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.456207 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a41e36f-ca5e-4e8a-b322-c801592f61a4-kube-api-access-vmr96\" (UniqueName: \"kubernetes.io/projected/2a41e36f-ca5e-4e8a-b322-c801592f61a4-kube-api-access-vmr96\") pod \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\" (UID: \"2a41e36f-ca5e-4e8a-b322-c801592f61a4\") " Nov 01 23:04:52.456930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.455231 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-openshift-ca" pod "2a41e36f-ca5e-4e8a-b322-c801592f61a4" (UID: "2a41e36f-ca5e-4e8a-b322-c801592f61a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:04:52.457292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.457255 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:52.459762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.459739 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a41e36f-ca5e-4e8a-b322-c801592f61a4-kube-api-access-vmr96" pod "2a41e36f-ca5e-4e8a-b322-c801592f61a4" (UID: "2a41e36f-ca5e-4e8a-b322-c801592f61a4"). InnerVolumeSpecName "kube-api-access-vmr96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:04:52.460486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.460461 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-xks-ca" pod "2a41e36f-ca5e-4e8a-b322-c801592f61a4" (UID: "2a41e36f-ca5e-4e8a-b322-c801592f61a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:04:52.558533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.558501 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a41e36f-ca5e-4e8a-b322-c801592f61a4-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:52.558533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.558530 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmr96\" (UniqueName: \"kubernetes.io/projected/2a41e36f-ca5e-4e8a-b322-c801592f61a4-kube-api-access-vmr96\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:04:52.986817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:52.986787 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:53.160850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:53.160768 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" Nov 01 23:04:53.160850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:53.160804 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc" event={"ID":"2a41e36f-ca5e-4e8a-b322-c801592f61a4","Type":"ContainerDied","Data":"5352782aa03739d094cc60e8250f4275e51be4a85fdbb0e8c7b1c86c28084449"} Nov 01 23:04:53.160850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:53.160850 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5352782aa03739d094cc60e8250f4275e51be4a85fdbb0e8c7b1c86c28084449" Nov 01 23:04:53.672791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:53.672757 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:54.076398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:54.076311 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:04:54.457376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:54.457339 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:55.075709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:55.075683 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:04:55.243184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:55.243143 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:55.891950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:55.891908 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:56.084240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:56.084206 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:04:56.084685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:56.084293 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:04:56.584535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:56.584501 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:57.361044 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:57.361010 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:58.034112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:58.034073 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:58.686016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:58.685980 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:59.025742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.025660 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9"] Nov 01 23:04:59.032292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.032266 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f04fd0b1-fd6p9"] Nov 01 23:04:59.090122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.090089 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:59.463637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.463565 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:59.576267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.576222 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:04:59.932486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.932458 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9"] Nov 01 23:04:59.933289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.933271 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2a41e36f-ca5e-4e8a-b322-c801592f61a4" containerName="revoke-keys" Nov 01 23:04:59.933344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.933291 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a41e36f-ca5e-4e8a-b322-c801592f61a4" containerName="revoke-keys" Nov 01 23:04:59.933451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.933442 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a41e36f-ca5e-4e8a-b322-c801592f61a4" containerName="revoke-keys" Nov 01 23:04:59.937681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.937662 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:04:59.953231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.953204 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9"] Nov 01 23:04:59.969309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.969278 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:04:59.970907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.970882 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:04:59.994106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:04:59.993915 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:00.087474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.087436 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjbm\" (UniqueName: \"kubernetes.io/projected/867cd23e-8b02-4690-8448-aa5eec26500a-kube-api-access-dxjbm\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.087699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.087515 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.087781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.087753 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.189063 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.188978 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.189244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.189161 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjbm\" (UniqueName: \"kubernetes.io/projected/867cd23e-8b02-4690-8448-aa5eec26500a-kube-api-access-dxjbm\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.189244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.189227 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.189760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.189735 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.191703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.191680 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.197914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.197893 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjbm\" (UniqueName: \"kubernetes.io/projected/867cd23e-8b02-4690-8448-aa5eec26500a-kube-api-access-dxjbm\") pod \"maas-api-revoke-sub-cf23f11c-mnmt9\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.253220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.253190 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:00.562354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.562306 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:00.720560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.720531 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9"] Nov 01 23:05:00.720901 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:05:00.720873 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867cd23e_8b02_4690_8448_aa5eec26500a.slice/crio-6a20ec206750ea45be77536e7b7656aa87ece985a26a80229f34b07b2d5c50cb WatchSource:0}: Error finding container 6a20ec206750ea45be77536e7b7656aa87ece985a26a80229f34b07b2d5c50cb: Status 404 returned error can't find the container with id 6a20ec206750ea45be77536e7b7656aa87ece985a26a80229f34b07b2d5c50cb Nov 01 23:05:00.739062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:00.739026 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cac79bc-72cd-4f75-9fb6-5c15879bdfef" path="/var/lib/kubelet/pods/5cac79bc-72cd-4f75-9fb6-5c15879bdfef/volumes" Nov 01 23:05:01.233371 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:01.233333 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" event={"ID":"867cd23e-8b02-4690-8448-aa5eec26500a","Type":"ContainerStarted","Data":"3b55d023f88d29a07c5dd995f9425b252396c845cf5f21c2e120af23fdc0a405"} Nov 01 23:05:01.233839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:01.233397 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" event={"ID":"867cd23e-8b02-4690-8448-aa5eec26500a","Type":"ContainerStarted","Data":"6a20ec206750ea45be77536e7b7656aa87ece985a26a80229f34b07b2d5c50cb"} Nov 01 23:05:01.257744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:01.257680 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" podStartSLOduration=2.257660612 podStartE2EDuration="2.257660612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:01.25272483 +0000 UTC m=+2466.680602798" watchObservedRunningTime="2026-11-01 23:05:01.257660612 +0000 UTC m=+2466.685538543" Nov 01 23:05:01.743180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:01.743142 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:01.953259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:01.953224 4981 scope.go:122] "RemoveContainer" containerID="ce3d3ffcb13ed5807ded4661ca543863ae443a764f57d51fb768e6456519e119" Nov 01 23:05:01.967552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:01.967530 4981 scope.go:122] "RemoveContainer" containerID="0f7a60c70f9c1c6f10ab89c4c6b33aea020365aa3906875a6f143f239442033e" Nov 01 23:05:02.023560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.023533 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254"] Nov 01 23:05:02.027828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.027804 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcdd9d5c-q9254"] Nov 01 23:05:02.137679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.137644 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:02.244263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.244171 4981 generic.go:356] "Generic (PLEG): container finished" podID="867cd23e-8b02-4690-8448-aa5eec26500a" containerID="3b55d023f88d29a07c5dd995f9425b252396c845cf5f21c2e120af23fdc0a405" exitCode=0 Nov 01 23:05:02.244263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.244257 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" event={"ID":"867cd23e-8b02-4690-8448-aa5eec26500a","Type":"ContainerDied","Data":"3b55d023f88d29a07c5dd995f9425b252396c845cf5f21c2e120af23fdc0a405"} Nov 01 23:05:02.602769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.602345 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:02.711613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.708039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:02.740502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.740469 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03fff2db-3e5f-42e6-b407-ded0679a0c1c" path="/var/lib/kubelet/pods/03fff2db-3e5f-42e6-b407-ded0679a0c1c/volumes" Nov 01 23:05:02.831402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:02.831371 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:03.534653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.521146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:03.619607 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.619555 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:03.656914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.656887 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:03.725241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.725208 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-xks-ca\") pod \"867cd23e-8b02-4690-8448-aa5eec26500a\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " Nov 01 23:05:03.725425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.725278 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-openshift-ca\") pod \"867cd23e-8b02-4690-8448-aa5eec26500a\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " Nov 01 23:05:03.725515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.725462 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867cd23e-8b02-4690-8448-aa5eec26500a-kube-api-access-dxjbm\" (UniqueName: \"kubernetes.io/projected/867cd23e-8b02-4690-8448-aa5eec26500a-kube-api-access-dxjbm\") pod \"867cd23e-8b02-4690-8448-aa5eec26500a\" (UID: \"867cd23e-8b02-4690-8448-aa5eec26500a\") " Nov 01 23:05:03.727107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.727081 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-openshift-ca" pod "867cd23e-8b02-4690-8448-aa5eec26500a" (UID: "867cd23e-8b02-4690-8448-aa5eec26500a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:05:03.739240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.736189 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867cd23e-8b02-4690-8448-aa5eec26500a-kube-api-access-dxjbm" pod "867cd23e-8b02-4690-8448-aa5eec26500a" (UID: "867cd23e-8b02-4690-8448-aa5eec26500a"). InnerVolumeSpecName "kube-api-access-dxjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:03.739458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.739401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:03.750505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.750473 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-xks-ca" pod "867cd23e-8b02-4690-8448-aa5eec26500a" (UID: "867cd23e-8b02-4690-8448-aa5eec26500a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:05:03.827081 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.826999 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:03.827081 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.827031 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867cd23e-8b02-4690-8448-aa5eec26500a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:03.827081 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.827046 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjbm\" (UniqueName: \"kubernetes.io/projected/867cd23e-8b02-4690-8448-aa5eec26500a-kube-api-access-dxjbm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:03.837415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.837373 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:03.996599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:03.993295 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:04.230625 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:04.229275 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:04.548658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:04.544479 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:04.563597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:04.558364 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" event={"ID":"867cd23e-8b02-4690-8448-aa5eec26500a","Type":"ContainerDied","Data":"6a20ec206750ea45be77536e7b7656aa87ece985a26a80229f34b07b2d5c50cb"} Nov 01 23:05:04.563597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:04.558505 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a20ec206750ea45be77536e7b7656aa87ece985a26a80229f34b07b2d5c50cb" Nov 01 23:05:04.563597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:04.558692 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9" Nov 01 23:05:04.899943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:04.899845 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:05.667464 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:05.667426 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:06.357098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:06.357066 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:07.288606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:07.288303 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:08.097446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:08.097406 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:09.038786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:09.038723 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:09.753990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:09.753961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:10.474109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:10.474066 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:11.189901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:11.189847 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:11.618345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:11.618246 4981 generic.go:356] "Generic (PLEG): container finished" podID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerID="c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492" exitCode=0 Nov 01 23:05:11.618345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:11.618327 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" event={"ID":"16cbedbb-ef98-4fc8-927b-a47634d5e428","Type":"ContainerDied","Data":"c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492"} Nov 01 23:05:11.618867 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:11.618821 4981 scope.go:122] "RemoveContainer" containerID="c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492" Nov 01 23:05:11.620836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:11.620809 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:05:11.620926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:11.620876 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:05:11.620926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:11.620907 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:05:12.055195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:12.055157 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:12.628744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:12.628703 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" event={"ID":"16cbedbb-ef98-4fc8-927b-a47634d5e428","Type":"ContainerStarted","Data":"55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6"} Nov 01 23:05:12.967626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:12.966265 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:12.997878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:12.997817 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:05:12.998924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:12.998847 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 01 23:05:12.999162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:12.999136 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 01 23:05:13.100413 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:13.100035 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:13.803806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:13.803771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:13.996613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:13.996039 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 01 23:05:13.996613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:13.996122 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 01 23:05:14.527642 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:05:14.527605 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:05:14.701072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:14.701008 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:14.999021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:14.998980 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 01 23:05:14.999468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:14.999042 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 01 23:05:15.098601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:15.098073 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:15.365599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:15.365484 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:05:15.595599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:15.595050 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:15.833400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:15.833368 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:15.998601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:15.995939 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 01 23:05:15.998601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:15.995998 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 01 23:05:16.702035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:16.701995 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:16.994443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:16.994348 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body= Nov 01 23:05:16.994726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:16.994695 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" Nov 01 23:05:17.437367 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:17.437334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:17.999706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:17.999676 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:05:18.000993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:18.000969 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:05:18.370022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:18.369940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:19.109856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:19.109818 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:19.946478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:19.946437 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:20.951566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:20.951534 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:21.421820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.421740 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:21.516669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.516599 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:21.517357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.517336 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv"] Nov 01 23:05:21.518205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.518186 4981 cpu_manager.go:414] "containerMap: removing container" podUID="867cd23e-8b02-4690-8448-aa5eec26500a" containerName="revoke-keys" Nov 01 23:05:21.518205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.518204 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867cd23e-8b02-4690-8448-aa5eec26500a" containerName="revoke-keys" Nov 01 23:05:21.518321 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.518314 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="867cd23e-8b02-4690-8448-aa5eec26500a" containerName="revoke-keys" Nov 01 23:05:21.522218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.522203 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.530496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.530473 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8e3381-dockercfg-x5j85\"" Nov 01 23:05:21.530665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.530597 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8e3381\"" Nov 01 23:05:21.533023 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.532999 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv"] Nov 01 23:05:21.549407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.549371 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p"] Nov 01 23:05:21.556963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.556938 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:21.563124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.563094 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p"] Nov 01 23:05:21.571214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.571190 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c3010-866b-4d89-8100-a08e10a7ea0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:21.571321 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.571287 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9n9d\" (UniqueName: \"kubernetes.io/projected/00e134e2-2bee-4b6d-b70b-82a8866688c5-kube-api-access-q9n9d\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.571367 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.571346 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktvf\" (UniqueName: \"kubernetes.io/projected/535c3010-866b-4d89-8100-a08e10a7ea0b-kube-api-access-hktvf\") pod \"payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:21.571405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.571375 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e134e2-2bee-4b6d-b70b-82a8866688c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.678598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.676348 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c3010-866b-4d89-8100-a08e10a7ea0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:21.678598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.676473 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9n9d\" (UniqueName: \"kubernetes.io/projected/00e134e2-2bee-4b6d-b70b-82a8866688c5-kube-api-access-q9n9d\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.678598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.676553 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktvf\" (UniqueName: \"kubernetes.io/projected/535c3010-866b-4d89-8100-a08e10a7ea0b-kube-api-access-hktvf\") pod \"payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:21.678598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.676617 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e134e2-2bee-4b6d-b70b-82a8866688c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.678598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.677669 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e134e2-2bee-4b6d-b70b-82a8866688c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.678598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.678398 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c3010-866b-4d89-8100-a08e10a7ea0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:21.724443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.724408 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9n9d\" (UniqueName: \"kubernetes.io/projected/00e134e2-2bee-4b6d-b70b-82a8866688c5-kube-api-access-q9n9d\") pod \"payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.769231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.769119 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktvf\" (UniqueName: \"kubernetes.io/projected/535c3010-866b-4d89-8100-a08e10a7ea0b-kube-api-access-hktvf\") pod \"payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:21.840914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.840866 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:21.877483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:21.877449 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:22.075035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.074965 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:22.138506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.138471 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv"] Nov 01 23:05:22.139521 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:05:22.139475 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e134e2_2bee_4b6d_b70b_82a8866688c5.slice/crio-cca11733de83c9aec54c1931aa4bea6c7a6474f0bb3179f0799836a144b2b01e WatchSource:0}: Error finding container cca11733de83c9aec54c1931aa4bea6c7a6474f0bb3179f0799836a144b2b01e: Status 404 returned error can't find the container with id cca11733de83c9aec54c1931aa4bea6c7a6474f0bb3179f0799836a144b2b01e Nov 01 23:05:22.175244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.175219 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:22.185562 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:05:22.185538 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535c3010_866b_4d89_8100_a08e10a7ea0b.slice/crio-e881ad170787e20e159fb70a75e0d59dc191e351ff3c8f29dc4ba55857f41666 WatchSource:0}: Error finding container e881ad170787e20e159fb70a75e0d59dc191e351ff3c8f29dc4ba55857f41666: Status 404 returned error can't find the container with id e881ad170787e20e159fb70a75e0d59dc191e351ff3c8f29dc4ba55857f41666 Nov 01 23:05:22.186743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.186723 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p"] Nov 01 23:05:22.322740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.322453 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:22.709374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.709325 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:22.800018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.799970 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" event={"ID":"00e134e2-2bee-4b6d-b70b-82a8866688c5","Type":"ContainerStarted","Data":"02e26adf8941becc587065d42a0812bb4f257bfef94114cf22ebd912c45fcb33"} Nov 01 23:05:22.800018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.800036 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" event={"ID":"00e134e2-2bee-4b6d-b70b-82a8866688c5","Type":"ContainerStarted","Data":"cca11733de83c9aec54c1931aa4bea6c7a6474f0bb3179f0799836a144b2b01e"} Nov 01 23:05:22.800018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.800080 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:22.802516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.802371 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" event={"ID":"535c3010-866b-4d89-8100-a08e10a7ea0b","Type":"ContainerStarted","Data":"91ad56dd3fa993ad6e6e130cb4d15acd7312820438ef58734cfbd827cfc4a814"} Nov 01 23:05:22.802516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.802441 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" event={"ID":"535c3010-866b-4d89-8100-a08e10a7ea0b","Type":"ContainerStarted","Data":"e881ad170787e20e159fb70a75e0d59dc191e351ff3c8f29dc4ba55857f41666"} Nov 01 23:05:22.802516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.802472 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:22.857160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.857116 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" podStartSLOduration=1.85710303 podStartE2EDuration="1.85710303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:22.852769843 +0000 UTC m=+2488.280647771" watchObservedRunningTime="2026-11-01 23:05:22.85710303 +0000 UTC m=+2488.284980958" Nov 01 23:05:22.894606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:22.894535 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" podStartSLOduration=1.8945173450000001 podStartE2EDuration="1.894517345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:22.891285114 +0000 UTC m=+2488.319163045" watchObservedRunningTime="2026-11-01 23:05:22.894517345 +0000 UTC m=+2488.322395278" Nov 01 23:05:23.217173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:23.217137 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:23.408599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:23.399283 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:23.506655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:23.506267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:24.069259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:24.069229 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:24.839404 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:24.839369 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:25.726780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:25.726744 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:26.769162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:26.769120 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:27.017561 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:27.017528 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:27.711519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:27.711483 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:28.075589 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:28.075492 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:28.428720 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:28.408147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:28.784643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:28.784517 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:29.713911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:29.713868 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:30.489506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:30.489469 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:31.270541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:31.270509 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:31.949488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:31.949451 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:32.688790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:32.688756 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:33.567900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:33.567866 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:33.882302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:33.882225 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:05:33.882784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:33.882655 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:05:34.261890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:34.261842 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:34.954300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:34.954262 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:35.694358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:35.694319 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:35.794611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:35.794564 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:36.107856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:36.107712 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:36.567040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:36.567010 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:36.717091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:36.717037 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:37.679990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:37.679962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:38.739257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:38.739222 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:39.307289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:39.307238 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:39.436110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:39.436072 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:39.892502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:39.892463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:40.521997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:40.521965 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:40.648867 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:40.648831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:41.304948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:41.304905 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:42.051980 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:42.051952 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:42.761001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:42.760969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:43.445521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:43.445486 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:44.273527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.273493 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt"] Nov 01 23:05:44.280297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.280268 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.285565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.285536 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt"] Nov 01 23:05:44.285777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.285744 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:05:44.286927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.286905 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:05:44.434232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.434200 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5ss\" (UniqueName: \"kubernetes.io/projected/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-kube-api-access-zt5ss\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.434446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.434397 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.434550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.434532 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.483460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.483427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:44.536359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.536262 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.543079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.543033 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.543255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.543147 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5ss\" (UniqueName: \"kubernetes.io/projected/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-kube-api-access-zt5ss\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.543687 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:05:44.536285 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:05:44.543899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.537352 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.553436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.553405 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.558604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.554656 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5ss\" (UniqueName: \"kubernetes.io/projected/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-kube-api-access-zt5ss\") pod \"maas-api-revoke-sub-21387493-9mwnt\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.622134 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.622104 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:44.914777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:44.914698 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt"] Nov 01 23:05:44.919180 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:05:44.919151 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f073c38_e91a_4d1e_bfde_1ae99f4d9fe8.slice/crio-5408c0d067d426028e2c501f85cff2acfd001c57ddf407193c6459cb0eca007e WatchSource:0}: Error finding container 5408c0d067d426028e2c501f85cff2acfd001c57ddf407193c6459cb0eca007e: Status 404 returned error can't find the container with id 5408c0d067d426028e2c501f85cff2acfd001c57ddf407193c6459cb0eca007e Nov 01 23:05:45.221878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:45.221836 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" event={"ID":"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8","Type":"ContainerStarted","Data":"9e73464e555ee859071b35cb510d62acb5d5903f50cb130bb6212a5948df69d6"} Nov 01 23:05:45.222007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:45.221910 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" event={"ID":"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8","Type":"ContainerStarted","Data":"5408c0d067d426028e2c501f85cff2acfd001c57ddf407193c6459cb0eca007e"} Nov 01 23:05:45.240232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:45.240188 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" podStartSLOduration=1.240174793 podStartE2EDuration="1.240174793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:45.239058786 +0000 UTC m=+2510.666936718" watchObservedRunningTime="2026-11-01 23:05:45.240174793 +0000 UTC m=+2510.668052724" Nov 01 23:05:45.371270 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:45.371233 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:46.227747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:46.227713 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:46.232625 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:46.232570 4981 generic.go:356] "Generic (PLEG): container finished" podID="1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" containerID="9e73464e555ee859071b35cb510d62acb5d5903f50cb130bb6212a5948df69d6" exitCode=0 Nov 01 23:05:46.232761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:46.232686 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" event={"ID":"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8","Type":"ContainerDied","Data":"9e73464e555ee859071b35cb510d62acb5d5903f50cb130bb6212a5948df69d6"} Nov 01 23:05:46.959998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:46.959969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:47.427535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.427506 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:47.603924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.603843 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-kube-api-access-zt5ss\" (UniqueName: \"kubernetes.io/projected/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-kube-api-access-zt5ss\") pod \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " Nov 01 23:05:47.603924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.603889 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-openshift-ca\") pod \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " Nov 01 23:05:47.604206 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.603955 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-xks-ca\") pod \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\" (UID: \"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8\") " Nov 01 23:05:47.604340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.604316 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-openshift-ca" pod "1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" (UID: "1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:05:47.604736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.604711 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:47.606152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.606122 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-xks-ca" pod "1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" (UID: "1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:05:47.606252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.606150 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-kube-api-access-zt5ss" pod "1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" (UID: "1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8"). InnerVolumeSpecName "kube-api-access-zt5ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:47.706162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.706128 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5ss\" (UniqueName: \"kubernetes.io/projected/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-kube-api-access-zt5ss\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:47.706162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.706156 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:47.878514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.878425 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:47.978239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:47.978195 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:48.295203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:48.295173 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" Nov 01 23:05:48.298115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:48.295225 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt" event={"ID":"1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8","Type":"ContainerDied","Data":"5408c0d067d426028e2c501f85cff2acfd001c57ddf407193c6459cb0eca007e"} Nov 01 23:05:48.298266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:48.298175 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5408c0d067d426028e2c501f85cff2acfd001c57ddf407193c6459cb0eca007e" Nov 01 23:05:48.308305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:48.308277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:48.406386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:48.406357 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:48.537782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:48.537748 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:48.819958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:48.819925 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:49.861086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:49.861057 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:50.576335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:50.576305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:51.265715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:51.265688 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:52.044749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:52.044702 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:52.762418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:52.762384 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:53.391595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:53.391548 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:54.117992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.117962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:54.560630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.560598 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh"] Nov 01 23:05:54.561421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.561404 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" containerName="revoke-keys" Nov 01 23:05:54.561421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.561421 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" containerName="revoke-keys" Nov 01 23:05:54.561566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.561557 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" containerName="revoke-keys" Nov 01 23:05:54.565439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.565415 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.573483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.571475 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:05:54.573483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.571629 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:05:54.578000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.577970 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh"] Nov 01 23:05:54.667038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.667003 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.667215 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.667117 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.667273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.667252 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnknk\" (UniqueName: \"kubernetes.io/projected/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-kube-api-access-cnknk\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.768330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.768300 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.768512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.768383 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.768512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.768454 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnknk\" (UniqueName: \"kubernetes.io/projected/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-kube-api-access-cnknk\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.769053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.769031 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.770618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.770595 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.777953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.777928 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnknk\" (UniqueName: \"kubernetes.io/projected/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-kube-api-access-cnknk\") pod \"maas-api-revoke-sub-d60ec40d-kbssh\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:54.876235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.876164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:54.885825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:54.885802 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:55.215059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:55.215023 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh"] Nov 01 23:05:55.215799 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:05:55.215773 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495deed8_b425_4e18_b6d5_6cf2f3c2b42a.slice/crio-a561519380153b1d2e6607d107a83b0626bfb872b81de6ac3dd57cd9661da139 WatchSource:0}: Error finding container a561519380153b1d2e6607d107a83b0626bfb872b81de6ac3dd57cd9661da139: Status 404 returned error can't find the container with id a561519380153b1d2e6607d107a83b0626bfb872b81de6ac3dd57cd9661da139 Nov 01 23:05:55.405274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:55.400452 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" event={"ID":"495deed8-b425-4e18-b6d5-6cf2f3c2b42a","Type":"ContainerStarted","Data":"a561519380153b1d2e6607d107a83b0626bfb872b81de6ac3dd57cd9661da139"} Nov 01 23:05:55.656742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:55.656701 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:56.270436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:56.270407 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:56.410290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:56.410200 4981 generic.go:356] "Generic (PLEG): container finished" podID="495deed8-b425-4e18-b6d5-6cf2f3c2b42a" containerID="650983d678dd5a39ace98eb587b45799138b31b9cb9c2cae78f3203e68904f4b" exitCode=0 Nov 01 23:05:56.410441 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:56.410281 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" event={"ID":"495deed8-b425-4e18-b6d5-6cf2f3c2b42a","Type":"ContainerDied","Data":"650983d678dd5a39ace98eb587b45799138b31b9cb9c2cae78f3203e68904f4b"} Nov 01 23:05:56.961635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:56.961569 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:57.598256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.598233 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:57.724335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.724304 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-kube-api-access-cnknk\" (UniqueName: \"kubernetes.io/projected/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-kube-api-access-cnknk\") pod \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " Nov 01 23:05:57.724507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.724399 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-xks-ca\") pod \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " Nov 01 23:05:57.724507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.724454 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-openshift-ca\") pod \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\" (UID: \"495deed8-b425-4e18-b6d5-6cf2f3c2b42a\") " Nov 01 23:05:57.724908 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.724875 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-openshift-ca" pod "495deed8-b425-4e18-b6d5-6cf2f3c2b42a" (UID: "495deed8-b425-4e18-b6d5-6cf2f3c2b42a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:05:57.725468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.725445 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:57.727077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.727044 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-kube-api-access-cnknk" pod "495deed8-b425-4e18-b6d5-6cf2f3c2b42a" (UID: "495deed8-b425-4e18-b6d5-6cf2f3c2b42a"). InnerVolumeSpecName "kube-api-access-cnknk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:05:57.727414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.727385 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-xks-ca" pod "495deed8-b425-4e18-b6d5-6cf2f3c2b42a" (UID: "495deed8-b425-4e18-b6d5-6cf2f3c2b42a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:05:57.787780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.787615 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:57.827264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.827229 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnknk\" (UniqueName: \"kubernetes.io/projected/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-kube-api-access-cnknk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:57.827264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:57.827260 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/495deed8-b425-4e18-b6d5-6cf2f3c2b42a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:05:58.427330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:58.427294 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" event={"ID":"495deed8-b425-4e18-b6d5-6cf2f3c2b42a","Type":"ContainerDied","Data":"a561519380153b1d2e6607d107a83b0626bfb872b81de6ac3dd57cd9661da139"} Nov 01 23:05:58.427510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:58.427350 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a561519380153b1d2e6607d107a83b0626bfb872b81de6ac3dd57cd9661da139" Nov 01 23:05:58.427510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:58.427307 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh" Nov 01 23:05:58.454120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:58.454098 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:59.275680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:59.275646 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:05:59.956247 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:05:59.956203 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:00.070454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:00.069793 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:00.766894 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:00.766858 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:01.488297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:01.488263 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:02.058814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:02.058783 4981 scope.go:122] "RemoveContainer" containerID="36710a7049e81c8fe1db5b83bc95e48f66275fe8a0a3485dba8c2088156bcbbd" Nov 01 23:06:02.070128 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:02.070103 4981 scope.go:122] "RemoveContainer" containerID="017066b741a6b9800fcc025b61aefa74360bf7682d02c0921b0b12e51d2fc332" Nov 01 23:06:02.081554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:02.081532 4981 scope.go:122] "RemoveContainer" containerID="ef50c3e89bd1a37c16fd38e26c287ac87a735a4c65b4bb1898a4fc83c6bc16ca" Nov 01 23:06:02.092461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:02.092435 4981 scope.go:122] "RemoveContainer" containerID="068218a54bcbeb75ee704ab8d3e2c0afc984d9972fcc5d430bdce8692062ae87" Nov 01 23:06:02.247422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:02.247389 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:02.885079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:02.885040 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:03.557728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:03.557696 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:04.274102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:04.274072 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:05.139173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:05.139043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:05.891343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:05.891296 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:06.289322 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.289286 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:06.418247 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.418028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:06.686804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.686771 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:06.763499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.763472 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28"] Nov 01 23:06:06.764323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.764304 4981 cpu_manager.go:414] "containerMap: removing container" podUID="495deed8-b425-4e18-b6d5-6cf2f3c2b42a" containerName="revoke-keys" Nov 01 23:06:06.764386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.764324 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495deed8-b425-4e18-b6d5-6cf2f3c2b42a" containerName="revoke-keys" Nov 01 23:06:06.764508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.764499 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="495deed8-b425-4e18-b6d5-6cf2f3c2b42a" containerName="revoke-keys" Nov 01 23:06:06.771015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.770995 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:06.777271 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.777246 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-85b1ad\"" Nov 01 23:06:06.777462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.777443 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-85b1ad-dockercfg-mwrgp\"" Nov 01 23:06:06.779258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.779236 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28"] Nov 01 23:06:06.868631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.859897 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:06.888601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.869142 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr429\" (UniqueName: \"kubernetes.io/projected/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-kube-api-access-kr429\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:06.993897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.993789 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:06.994461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.994430 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr429\" (UniqueName: \"kubernetes.io/projected/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-kube-api-access-kr429\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:06.995154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:06.995124 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:07.022887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.022857 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c"] Nov 01 23:06:07.040772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.040738 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c"] Nov 01 23:06:07.040956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.040856 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.041077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.041055 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr429\" (UniqueName: \"kubernetes.io/projected/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-kube-api-access-kr429\") pod \"payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:07.090952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.090927 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:07.197009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.196981 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqsp8\" (UniqueName: \"kubernetes.io/projected/8e865675-b66f-4bc2-af3d-98477f386c8e-kube-api-access-qqsp8\") pod \"payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.197161 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.197020 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865675-b66f-4bc2-af3d-98477f386c8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.298406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.298315 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqsp8\" (UniqueName: \"kubernetes.io/projected/8e865675-b66f-4bc2-af3d-98477f386c8e-kube-api-access-qqsp8\") pod \"payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.298406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.298361 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865675-b66f-4bc2-af3d-98477f386c8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.299201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.299179 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865675-b66f-4bc2-af3d-98477f386c8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.308047 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.308021 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqsp8\" (UniqueName: \"kubernetes.io/projected/8e865675-b66f-4bc2-af3d-98477f386c8e-kube-api-access-qqsp8\") pod \"payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.309752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.309725 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:07.379447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.379420 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:07.382684 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:07.382657 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a03d5b_6491_4f44_9ea6_dc6f1e612ca0.slice/crio-b4f0a5599c5dd53e08729038f35ce45f659a5423517268c10f762e2736e56718 WatchSource:0}: Error finding container b4f0a5599c5dd53e08729038f35ce45f659a5423517268c10f762e2736e56718: Status 404 returned error can't find the container with id b4f0a5599c5dd53e08729038f35ce45f659a5423517268c10f762e2736e56718 Nov 01 23:06:07.382684 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.382663 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28"] Nov 01 23:06:07.629883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.629788 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" event={"ID":"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0","Type":"ContainerStarted","Data":"3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a"} Nov 01 23:06:07.629883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.629858 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:07.629883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.629880 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" event={"ID":"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0","Type":"ContainerStarted","Data":"b4f0a5599c5dd53e08729038f35ce45f659a5423517268c10f762e2736e56718"} Nov 01 23:06:07.654053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.654008 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" podStartSLOduration=1.653995159 podStartE2EDuration="1.653995159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:07.652797502 +0000 UTC m=+2533.080675434" watchObservedRunningTime="2026-11-01 23:06:07.653995159 +0000 UTC m=+2533.081873130" Nov 01 23:06:07.670324 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:07.670297 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e865675_b66f_4bc2_af3d_98477f386c8e.slice/crio-b4d85da1dac8d358154092a47b4d8fe51bda085f7e9e8edb44b47410568ed82d WatchSource:0}: Error finding container b4d85da1dac8d358154092a47b4d8fe51bda085f7e9e8edb44b47410568ed82d: Status 404 returned error can't find the container with id b4d85da1dac8d358154092a47b4d8fe51bda085f7e9e8edb44b47410568ed82d Nov 01 23:06:07.670598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:07.670563 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c"] Nov 01 23:06:08.092049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:08.091996 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:08.653538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:08.653495 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" event={"ID":"8e865675-b66f-4bc2-af3d-98477f386c8e","Type":"ContainerStarted","Data":"bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66"} Nov 01 23:06:08.654087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:08.653608 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" event={"ID":"8e865675-b66f-4bc2-af3d-98477f386c8e","Type":"ContainerStarted","Data":"b4d85da1dac8d358154092a47b4d8fe51bda085f7e9e8edb44b47410568ed82d"} Nov 01 23:06:08.654087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:08.653742 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:08.674359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:08.674299 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" podStartSLOduration=2.67428305 podStartE2EDuration="2.67428305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:08.67086703 +0000 UTC m=+2534.098744957" watchObservedRunningTime="2026-11-01 23:06:08.67428305 +0000 UTC m=+2534.102160987" Nov 01 23:06:08.925961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:08.925928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:09.806231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:09.806193 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:10.671743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:10.671704 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:11.439142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:11.439110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:11.873649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:11.873539 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2"] Nov 01 23:06:11.873832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:11.873780 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" containerID="cri-o://d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967" gracePeriod=30 Nov 01 23:06:12.244442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:12.244411 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:12.521030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:12.520939 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:12.619388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:12.619358 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:12.789148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:12.789057 4981 patch_prober.go:29] interesting pod/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:06:12.789406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:12.789378 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:06:12.792946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:12.792919 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:13.218651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:13.218568 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:14.113712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:14.113674 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:14.526879 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:06:14.526843 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:06:14.815130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:14.815053 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:14.982250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:14.982216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:15.256665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:15.256631 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:15.800735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:15.800690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:16.465490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:16.465457 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:17.173295 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.173261 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp"] Nov 01 23:06:17.178488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.178468 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.189257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.188653 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-075771-openshift-default-dockercfg-ht2bd\"" Nov 01 23:06:17.192834 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.192809 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp"] Nov 01 23:06:17.203840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.203811 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:17.221423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.221396 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 23:06:17.243090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.241569 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-podinfo\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.251781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.251740 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-token\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-token\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.252524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.252501 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad33ddb9-b1b9-4b1f-9691-607def69c690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad33ddb9-b1b9-4b1f-9691-607def69c690-istiod-ca-cert\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.253365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.253089 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-envoy\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.253568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.253549 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-credential-socket\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.253870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.253842 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-certs\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.254228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.254206 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-socket\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.254426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.254408 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-kube-api-access-cgt4f\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-kube-api-access-cgt4f\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.254840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.254797 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-data\") pod \"ad33ddb9-b1b9-4b1f-9691-607def69c690\" (UID: \"ad33ddb9-b1b9-4b1f-9691-607def69c690\") " Nov 01 23:06:17.255280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.255251 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.255432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.255413 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.255649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.255629 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.255849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.255830 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.256210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.256189 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.256383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.256365 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttfg5\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-kube-api-access-ttfg5\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.256544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.256527 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.256708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.256690 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.256853 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.256837 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.257262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.253017 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad33ddb9-b1b9-4b1f-9691-607def69c690-istiod-ca-cert" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:17.257596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.257558 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-credential-socket" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:17.257931 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.257909 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-certs" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:17.258241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.258221 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-socket" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:17.261148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.261127 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-data" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:17.264992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.264964 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-podinfo" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:06:17.293651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.289839 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-kube-api-access-cgt4f" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "kube-api-access-cgt4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:17.293651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.290033 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-token" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:17.312234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.297133 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-envoy" pod "ad33ddb9-b1b9-4b1f-9691-607def69c690" (UID: "ad33ddb9-b1b9-4b1f-9691-607def69c690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.358552 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359165 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359235 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359259 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359434 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359482 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttfg5\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-kube-api-access-ttfg5\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359526 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359552 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359092 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-credential-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.359856 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-socket\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.360262 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-certs\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.360451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.360426 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istiod-ca-cert\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.361312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.360656 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.361312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.360938 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-data\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362298 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362331 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgt4f\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-kube-api-access-cgt4f\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362348 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362362 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362375 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362387 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad33ddb9-b1b9-4b1f-9691-607def69c690-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362401 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362415 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.362623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362428 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad33ddb9-b1b9-4b1f-9691-607def69c690-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:17.363031 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.362754 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-envoy\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.363173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.363148 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-podinfo\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.370140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.370109 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-token\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.370965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.370942 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttfg5\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-kube-api-access-ttfg5\") pod \"e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.665897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.665861 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:17.830769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.830739 4981 generic.go:356] "Generic (PLEG): container finished" podID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerID="d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967" exitCode=0 Nov 01 23:06:17.830927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.830800 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" Nov 01 23:06:17.830927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.830818 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" event={"ID":"ad33ddb9-b1b9-4b1f-9691-607def69c690","Type":"ContainerDied","Data":"d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967"} Nov 01 23:06:17.830927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.830873 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2" event={"ID":"ad33ddb9-b1b9-4b1f-9691-607def69c690","Type":"ContainerDied","Data":"74e4fee70b43f0728360b6fe5d4de36a6c42568e163a9ad6d82687ea4668efe9"} Nov 01 23:06:17.830927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.830895 4981 scope.go:122] "RemoveContainer" containerID="d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967" Nov 01 23:06:17.842638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.842619 4981 scope.go:122] "RemoveContainer" containerID="d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a" Nov 01 23:06:17.853471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.853432 4981 scope.go:122] "RemoveContainer" containerID="d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967" Nov 01 23:06:17.854745 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:06:17.854716 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967\": container with ID starting with d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967 not found: ID does not exist" containerID="d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967" Nov 01 23:06:17.854862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.854754 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967"} err="failed to get container status \"d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967\": rpc error: code = NotFound desc = could not find container \"d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967\": container with ID starting with d972643b03ec9ba92a1ca8637b5472154caf6c64b7a1ff1b79b854f315bc1967 not found: ID does not exist" Nov 01 23:06:17.854862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.854779 4981 scope.go:122] "RemoveContainer" containerID="d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a" Nov 01 23:06:17.855103 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:06:17.855073 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a\": container with ID starting with d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a not found: ID does not exist" containerID="d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a" Nov 01 23:06:17.855162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.855111 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a"} err="failed to get container status \"d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a\": rpc error: code = NotFound desc = could not find container \"d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a\": container with ID starting with d9381a55576534f6d93ce00067169b85c05d298121f9d5a33fd7205efe32477a not found: ID does not exist" Nov 01 23:06:17.858454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.858432 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2"] Nov 01 23:06:17.862853 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.862833 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9ad303-openshift-default-55d6688c45-9zdh2"] Nov 01 23:06:17.896133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.896107 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:17.978605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.975218 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp"] Nov 01 23:06:17.981801 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:17.981770 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67106f5_6491_4f06_a60e_c5cf7d0d6c64.slice/crio-174a301a84f8336319eb98075b824d40abde5e13bb4efe09a862ca6246977ebd WatchSource:0}: Error finding container 174a301a84f8336319eb98075b824d40abde5e13bb4efe09a862ca6246977ebd: Status 404 returned error can't find the container with id 174a301a84f8336319eb98075b824d40abde5e13bb4efe09a862ca6246977ebd Nov 01 23:06:17.985108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.985070 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:06:17.985218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.985161 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:06:17.985218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:17.985208 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:06:18.656041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:18.656014 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:06:18.663309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:18.663282 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:18.739543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:18.739509 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" path="/var/lib/kubelet/pods/ad33ddb9-b1b9-4b1f-9691-607def69c690/volumes" Nov 01 23:06:18.842187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:18.842153 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" event={"ID":"b67106f5-6491-4f06-a60e-c5cf7d0d6c64","Type":"ContainerStarted","Data":"eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb"} Nov 01 23:06:18.842371 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:18.842204 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" event={"ID":"b67106f5-6491-4f06-a60e-c5cf7d0d6c64","Type":"ContainerStarted","Data":"174a301a84f8336319eb98075b824d40abde5e13bb4efe09a862ca6246977ebd"} Nov 01 23:06:18.868596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:18.868536 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podStartSLOduration=1.86852186 podStartE2EDuration="1.86852186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:18.866383227 +0000 UTC m=+2544.294261172" watchObservedRunningTime="2026-11-01 23:06:18.86852186 +0000 UTC m=+2544.296399791" Nov 01 23:06:19.600014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:19.599971 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:19.666029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:19.665992 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:19.668046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:19.668022 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 23:06:19.668203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:19.668105 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 23:06:19.701766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:19.701729 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:06:20.499785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:20.499742 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:20.667198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:20.667162 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 23:06:20.667381 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:20.667237 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 23:06:21.258703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:21.258672 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:21.666952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:21.666870 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 23:06:21.667315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:21.666945 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 23:06:21.970540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:21.970507 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:22.681599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:22.676083 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 23:06:22.681599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:22.676179 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 23:06:22.934049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:22.933948 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:23.666491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:23.666456 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 23:06:23.666741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:23.666523 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 23:06:23.699766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:23.699734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:24.350593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:24.350551 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:24.666619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:24.666524 4981 patch_prober.go:29] interesting pod/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 23:06:24.666776 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:24.666609 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 23:06:25.110015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.109980 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:25.207695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.207662 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:25.334917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.334868 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:25.453460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.453421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:25.672489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.672461 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:25.672978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.672955 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:25.673700 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.673680 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:06:25.876620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.876532 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz"] Nov 01 23:06:25.877324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.877306 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" Nov 01 23:06:25.877377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.877326 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" Nov 01 23:06:25.877377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.877358 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" Nov 01 23:06:25.877377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.877363 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" Nov 01 23:06:25.877513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.877504 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" Nov 01 23:06:25.877554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.877519 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad33ddb9-b1b9-4b1f-9691-607def69c690" containerName="istio-proxy" Nov 01 23:06:25.881458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.881439 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:25.897768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.897744 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz"] Nov 01 23:06:25.901397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.901364 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:06:25.901513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.901402 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:06:25.973541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.973508 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:25.973747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.973733 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:25.973794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:25.973778 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6x5\" (UniqueName: \"kubernetes.io/projected/6743b759-2915-4f26-8f10-d61662bb2a8a-kube-api-access-sh6x5\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.075415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.075378 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.075629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.075608 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.075720 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.075684 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6x5\" (UniqueName: \"kubernetes.io/projected/6743b759-2915-4f26-8f10-d61662bb2a8a-kube-api-access-sh6x5\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.076066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.076048 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.078030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.078005 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.087003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.086980 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6x5\" (UniqueName: \"kubernetes.io/projected/6743b759-2915-4f26-8f10-d61662bb2a8a-kube-api-access-sh6x5\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.199761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.199735 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:26.226084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.226061 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:26.507166 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:26.507133 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6743b759_2915_4f26_8f10_d61662bb2a8a.slice/crio-5291324f480df3f27af24ca76487a352fb373ec9620bb08f9e07391fdd5f7640 WatchSource:0}: Error finding container 5291324f480df3f27af24ca76487a352fb373ec9620bb08f9e07391fdd5f7640: Status 404 returned error can't find the container with id 5291324f480df3f27af24ca76487a352fb373ec9620bb08f9e07391fdd5f7640 Nov 01 23:06:26.507797 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.507774 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz"] Nov 01 23:06:26.936956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.936920 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" event={"ID":"6743b759-2915-4f26-8f10-d61662bb2a8a","Type":"ContainerStarted","Data":"5697bf827521e8bc098ff381498d633d5c9f26496697645707bddf38cb5198d6"} Nov 01 23:06:26.937168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.936975 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" event={"ID":"6743b759-2915-4f26-8f10-d61662bb2a8a","Type":"ContainerStarted","Data":"5291324f480df3f27af24ca76487a352fb373ec9620bb08f9e07391fdd5f7640"} Nov 01 23:06:26.956387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:26.956333 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" podStartSLOduration=1.956319203 podStartE2EDuration="1.956319203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:26.952795472 +0000 UTC m=+2552.380673403" watchObservedRunningTime="2026-11-01 23:06:26.956319203 +0000 UTC m=+2552.384197134" Nov 01 23:06:27.055059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:27.054983 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:27.354596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:27.354511 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:27.946543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:27.946512 4981 generic.go:356] "Generic (PLEG): container finished" podID="6743b759-2915-4f26-8f10-d61662bb2a8a" containerID="5697bf827521e8bc098ff381498d633d5c9f26496697645707bddf38cb5198d6" exitCode=0 Nov 01 23:06:27.946760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:27.946620 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" event={"ID":"6743b759-2915-4f26-8f10-d61662bb2a8a","Type":"ContainerDied","Data":"5697bf827521e8bc098ff381498d633d5c9f26496697645707bddf38cb5198d6"} Nov 01 23:06:28.061701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:28.061665 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:28.866989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:28.866953 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:29.155727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.155700 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:29.255275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.255237 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-openshift-ca\") pod \"6743b759-2915-4f26-8f10-d61662bb2a8a\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " Nov 01 23:06:29.255514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.255492 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-xks-ca\") pod \"6743b759-2915-4f26-8f10-d61662bb2a8a\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " Nov 01 23:06:29.255617 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.255530 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6743b759-2915-4f26-8f10-d61662bb2a8a-kube-api-access-sh6x5\" (UniqueName: \"kubernetes.io/projected/6743b759-2915-4f26-8f10-d61662bb2a8a-kube-api-access-sh6x5\") pod \"6743b759-2915-4f26-8f10-d61662bb2a8a\" (UID: \"6743b759-2915-4f26-8f10-d61662bb2a8a\") " Nov 01 23:06:29.255678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.255640 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-openshift-ca" pod "6743b759-2915-4f26-8f10-d61662bb2a8a" (UID: "6743b759-2915-4f26-8f10-d61662bb2a8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:29.256256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.256233 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:29.257770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.257748 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-xks-ca" pod "6743b759-2915-4f26-8f10-d61662bb2a8a" (UID: "6743b759-2915-4f26-8f10-d61662bb2a8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:29.258406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.258381 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6743b759-2915-4f26-8f10-d61662bb2a8a-kube-api-access-sh6x5" pod "6743b759-2915-4f26-8f10-d61662bb2a8a" (UID: "6743b759-2915-4f26-8f10-d61662bb2a8a"). InnerVolumeSpecName "kube-api-access-sh6x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:29.357884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.357842 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6743b759-2915-4f26-8f10-d61662bb2a8a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:29.357884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.357881 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6x5\" (UniqueName: \"kubernetes.io/projected/6743b759-2915-4f26-8f10-d61662bb2a8a-kube-api-access-sh6x5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:29.657989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.657942 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:29.976778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.976749 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" Nov 01 23:06:29.977303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.976784 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz" event={"ID":"6743b759-2915-4f26-8f10-d61662bb2a8a","Type":"ContainerDied","Data":"5291324f480df3f27af24ca76487a352fb373ec9620bb08f9e07391fdd5f7640"} Nov 01 23:06:29.977303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:29.976833 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5291324f480df3f27af24ca76487a352fb373ec9620bb08f9e07391fdd5f7640" Nov 01 23:06:30.369693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:30.369601 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:31.193139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.193108 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:31.662700 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.662518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:31.757378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.757343 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6"] Nov 01 23:06:31.758188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.758171 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6743b759-2915-4f26-8f10-d61662bb2a8a" containerName="revoke-keys" Nov 01 23:06:31.758255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.758189 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6743b759-2915-4f26-8f10-d61662bb2a8a" containerName="revoke-keys" Nov 01 23:06:31.758346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.758336 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6743b759-2915-4f26-8f10-d61662bb2a8a" containerName="revoke-keys" Nov 01 23:06:31.759948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.759921 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:31.762634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.762612 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:31.772870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.772462 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c00bbf\"" Nov 01 23:06:31.775197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.772616 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c00bbf-dockercfg-6dgh5\"" Nov 01 23:06:31.775197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.773072 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh"] Nov 01 23:06:31.780738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.780719 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6"] Nov 01 23:06:31.780831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.780801 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:31.798484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.798463 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh"] Nov 01 23:06:31.800002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.799976 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnj8w\" (UniqueName: \"kubernetes.io/projected/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-kube-api-access-mnj8w\") pod \"payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:31.800341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.800317 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:31.800628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.800567 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbec9e08-85fc-454b-b7ac-056410241350-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:31.800745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.800669 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptjt\" (UniqueName: \"kubernetes.io/projected/dbec9e08-85fc-454b-b7ac-056410241350-kube-api-access-wptjt\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:31.913140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.913043 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnj8w\" (UniqueName: \"kubernetes.io/projected/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-kube-api-access-mnj8w\") pod \"payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:31.913535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.913508 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:31.913803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.913782 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbec9e08-85fc-454b-b7ac-056410241350-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:31.913994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.913976 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptjt\" (UniqueName: \"kubernetes.io/projected/dbec9e08-85fc-454b-b7ac-056410241350-kube-api-access-wptjt\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:31.915990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.915969 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:31.917010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.916971 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbec9e08-85fc-454b-b7ac-056410241350-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:31.960429 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.960382 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnj8w\" (UniqueName: \"kubernetes.io/projected/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-kube-api-access-mnj8w\") pod \"payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:31.961364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:31.961327 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptjt\" (UniqueName: \"kubernetes.io/projected/dbec9e08-85fc-454b-b7ac-056410241350-kube-api-access-wptjt\") pod \"payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:32.081647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:32.081617 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:32.116397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:32.114943 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:32.267190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:32.267155 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:32.504699 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:32.504659 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbec9e08_85fc_454b_b7ac_056410241350.slice/crio-b63988de70931d3c214a22d1a5a396ffce8c0fdb19910d2768720ab16391cbfd WatchSource:0}: Error finding container b63988de70931d3c214a22d1a5a396ffce8c0fdb19910d2768720ab16391cbfd: Status 404 returned error can't find the container with id b63988de70931d3c214a22d1a5a396ffce8c0fdb19910d2768720ab16391cbfd Nov 01 23:06:32.515754 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:32.513126 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4e9cb9_84dc_43fb_9fc1_02a38671fbeb.slice/crio-593c9ff2b7a8d88d4db7596b9732524c5713d30be6c28170620d04c93990a1db WatchSource:0}: Error finding container 593c9ff2b7a8d88d4db7596b9732524c5713d30be6c28170620d04c93990a1db: Status 404 returned error can't find the container with id 593c9ff2b7a8d88d4db7596b9732524c5713d30be6c28170620d04c93990a1db Nov 01 23:06:32.519904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:32.519822 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6"] Nov 01 23:06:32.529497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:32.529471 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh"] Nov 01 23:06:33.039082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.039042 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" event={"ID":"dbec9e08-85fc-454b-b7ac-056410241350","Type":"ContainerStarted","Data":"dc3ce20d40bc812d239b4940d2f1b903255cb54126ea066dd7057e320fa78a16"} Nov 01 23:06:33.039301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.039104 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" event={"ID":"dbec9e08-85fc-454b-b7ac-056410241350","Type":"ContainerStarted","Data":"b63988de70931d3c214a22d1a5a396ffce8c0fdb19910d2768720ab16391cbfd"} Nov 01 23:06:33.039301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.039185 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:33.040942 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.040916 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" event={"ID":"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb","Type":"ContainerStarted","Data":"b05646d4fa5f925e31e68c90d30e8f995b180e99733e238762f4fdc041b04286"} Nov 01 23:06:33.041092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.040969 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" event={"ID":"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb","Type":"ContainerStarted","Data":"593c9ff2b7a8d88d4db7596b9732524c5713d30be6c28170620d04c93990a1db"} Nov 01 23:06:33.041092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.041014 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:33.061609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.061539 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" podStartSLOduration=2.061527479 podStartE2EDuration="2.061527479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:33.056007631 +0000 UTC m=+2558.483885562" watchObservedRunningTime="2026-11-01 23:06:33.061527479 +0000 UTC m=+2558.489405410" Nov 01 23:06:33.077563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.077520 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" podStartSLOduration=2.077508701 podStartE2EDuration="2.077508701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:33.072998498 +0000 UTC m=+2558.500876430" watchObservedRunningTime="2026-11-01 23:06:33.077508701 +0000 UTC m=+2558.505386632" Nov 01 23:06:33.188386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.188347 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:33.904596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:33.902330 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:34.593601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:34.590490 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:35.322670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:35.322641 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:36.102190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:36.102155 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:37.165180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:37.165147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:38.069892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:38.069857 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:38.765163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:38.765136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:39.547265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:39.547235 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:40.240622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:40.240588 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:40.906961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:40.906927 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:41.569443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:41.569412 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:42.342639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:42.342609 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:42.990663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:42.990634 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:43.136495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.136457 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:43.452217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.452181 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5"] Nov 01 23:06:43.460478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.460445 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.464399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.464373 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5"] Nov 01 23:06:43.465303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.465285 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:06:43.465509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.465373 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:06:43.577772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.577738 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.577956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.577789 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.578050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.578028 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6k9v\" (UniqueName: \"kubernetes.io/projected/5a8d29e3-8c54-408d-9371-b0f2267b3908-kube-api-access-t6k9v\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.678996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.678962 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6k9v\" (UniqueName: \"kubernetes.io/projected/5a8d29e3-8c54-408d-9371-b0f2267b3908-kube-api-access-t6k9v\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.679165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.679119 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.679241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.679172 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.679786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.679763 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.681481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.681457 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.687235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.687214 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6k9v\" (UniqueName: \"kubernetes.io/projected/5a8d29e3-8c54-408d-9371-b0f2267b3908-kube-api-access-t6k9v\") pod \"maas-api-revoke-sub-7cb1fc58-9dcl5\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.783687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.783616 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:43.916097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:43.916043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:44.065086 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:44.065056 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8d29e3_8c54_408d_9371_b0f2267b3908.slice/crio-85c9eef9762ccaef871c4918bfcdde2e328f1f6a0618caf722be508900a6a640 WatchSource:0}: Error finding container 85c9eef9762ccaef871c4918bfcdde2e328f1f6a0618caf722be508900a6a640: Status 404 returned error can't find the container with id 85c9eef9762ccaef871c4918bfcdde2e328f1f6a0618caf722be508900a6a640 Nov 01 23:06:44.065522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:44.065321 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5"] Nov 01 23:06:44.118057 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:44.118033 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:06:44.118177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:44.118099 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:06:44.214874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:44.214834 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" event={"ID":"5a8d29e3-8c54-408d-9371-b0f2267b3908","Type":"ContainerStarted","Data":"85c9eef9762ccaef871c4918bfcdde2e328f1f6a0618caf722be508900a6a640"} Nov 01 23:06:44.524266 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:06:44.524224 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:06:44.690012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:44.689980 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:45.182815 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.182784 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:45.225946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.225915 4981 generic.go:356] "Generic (PLEG): container finished" podID="5a8d29e3-8c54-408d-9371-b0f2267b3908" containerID="0b5bac2aa770efc732ef1d5075afa97c900c518c39a8f8e6c3acecb2078796f4" exitCode=0 Nov 01 23:06:45.226127 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.225998 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" event={"ID":"5a8d29e3-8c54-408d-9371-b0f2267b3908","Type":"ContainerDied","Data":"0b5bac2aa770efc732ef1d5075afa97c900c518c39a8f8e6c3acecb2078796f4"} Nov 01 23:06:45.272740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.272671 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f"] Nov 01 23:06:45.273661 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.273636 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:45.277488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.277469 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.282194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.282152 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-00185d-dockercfg-s77lt\"" Nov 01 23:06:45.282325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.282265 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-00185d\"" Nov 01 23:06:45.286593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.286560 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f"] Nov 01 23:06:45.327620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.324310 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc"] Nov 01 23:06:45.331777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.331743 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfxw\" (UniqueName: \"kubernetes.io/projected/f2525914-86ca-4583-8478-e185ad487b36-kube-api-access-6vfxw\") pod \"payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.335632 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.331992 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2525914-86ca-4583-8478-e185ad487b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.339807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.339784 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc"] Nov 01 23:06:45.339954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.339889 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:45.494253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.480922 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfxw\" (UniqueName: \"kubernetes.io/projected/f2525914-86ca-4583-8478-e185ad487b36-kube-api-access-6vfxw\") pod \"payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.494884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.494847 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2525914-86ca-4583-8478-e185ad487b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.497105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.496979 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2525914-86ca-4583-8478-e185ad487b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.578664 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.578411 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfxw\" (UniqueName: \"kubernetes.io/projected/f2525914-86ca-4583-8478-e185ad487b36-kube-api-access-6vfxw\") pod \"payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.595637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.595362 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:45.596604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.596561 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8ede17-ef50-4622-8ece-b80e35712dfd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:45.596751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.596644 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7kw\" (UniqueName: \"kubernetes.io/projected/9a8ede17-ef50-4622-8ece-b80e35712dfd-kube-api-access-7l7kw\") pod \"payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:45.698727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.698689 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8ede17-ef50-4622-8ece-b80e35712dfd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:45.698905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.698752 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7kw\" (UniqueName: \"kubernetes.io/projected/9a8ede17-ef50-4622-8ece-b80e35712dfd-kube-api-access-7l7kw\") pod \"payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:45.699880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.699859 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8ede17-ef50-4622-8ece-b80e35712dfd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:45.715447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.715422 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7kw\" (UniqueName: \"kubernetes.io/projected/9a8ede17-ef50-4622-8ece-b80e35712dfd-kube-api-access-7l7kw\") pod \"payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:45.928181 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:45.928148 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2525914_86ca_4583_8478_e185ad487b36.slice/crio-390d2404d5add9b262af047317f6be5fd10a75f63f6d08285c9d175bf936fd31 WatchSource:0}: Error finding container 390d2404d5add9b262af047317f6be5fd10a75f63f6d08285c9d175bf936fd31: Status 404 returned error can't find the container with id 390d2404d5add9b262af047317f6be5fd10a75f63f6d08285c9d175bf936fd31 Nov 01 23:06:45.932172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.932133 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:45.935115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.935094 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f"] Nov 01 23:06:45.960222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:45.960198 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:46.237326 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.237293 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" event={"ID":"f2525914-86ca-4583-8478-e185ad487b36","Type":"ContainerStarted","Data":"0c49f1be3186ba0c7da5ba7e99d44b7b069161446398dc730a8e5168465daf43"} Nov 01 23:06:46.237826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.237354 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" event={"ID":"f2525914-86ca-4583-8478-e185ad487b36","Type":"ContainerStarted","Data":"390d2404d5add9b262af047317f6be5fd10a75f63f6d08285c9d175bf936fd31"} Nov 01 23:06:46.237826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.237385 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:46.256432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.256402 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc"] Nov 01 23:06:46.258088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.257988 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" podStartSLOduration=1.257970993 podStartE2EDuration="1.257970993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:46.254924543 +0000 UTC m=+2571.682802475" watchObservedRunningTime="2026-11-01 23:06:46.257970993 +0000 UTC m=+2571.685848925" Nov 01 23:06:46.263713 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:46.263682 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8ede17_ef50_4622_8ece_b80e35712dfd.slice/crio-ee47fc69e0f44bd901137d60e428f28063846a4468e70e6f8022e44e80485dc2 WatchSource:0}: Error finding container ee47fc69e0f44bd901137d60e428f28063846a4468e70e6f8022e44e80485dc2: Status 404 returned error can't find the container with id ee47fc69e0f44bd901137d60e428f28063846a4468e70e6f8022e44e80485dc2 Nov 01 23:06:46.469173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.469146 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:46.515687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.513744 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-openshift-ca\") pod \"5a8d29e3-8c54-408d-9371-b0f2267b3908\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " Nov 01 23:06:46.515687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.513912 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a8d29e3-8c54-408d-9371-b0f2267b3908-kube-api-access-t6k9v\" (UniqueName: \"kubernetes.io/projected/5a8d29e3-8c54-408d-9371-b0f2267b3908-kube-api-access-t6k9v\") pod \"5a8d29e3-8c54-408d-9371-b0f2267b3908\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " Nov 01 23:06:46.515687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.514179 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-xks-ca\") pod \"5a8d29e3-8c54-408d-9371-b0f2267b3908\" (UID: \"5a8d29e3-8c54-408d-9371-b0f2267b3908\") " Nov 01 23:06:46.515687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.514200 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-openshift-ca" pod "5a8d29e3-8c54-408d-9371-b0f2267b3908" (UID: "5a8d29e3-8c54-408d-9371-b0f2267b3908"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:46.515687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.515080 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:46.517616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.517560 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-xks-ca" pod "5a8d29e3-8c54-408d-9371-b0f2267b3908" (UID: "5a8d29e3-8c54-408d-9371-b0f2267b3908"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:46.519655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.519630 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8d29e3-8c54-408d-9371-b0f2267b3908-kube-api-access-t6k9v" pod "5a8d29e3-8c54-408d-9371-b0f2267b3908" (UID: "5a8d29e3-8c54-408d-9371-b0f2267b3908"). InnerVolumeSpecName "kube-api-access-t6k9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:46.573935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.573893 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:46.616462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.616426 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a8d29e3-8c54-408d-9371-b0f2267b3908-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:46.616462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:46.616464 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6k9v\" (UniqueName: \"kubernetes.io/projected/5a8d29e3-8c54-408d-9371-b0f2267b3908-kube-api-access-t6k9v\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:47.247626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.247541 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" event={"ID":"5a8d29e3-8c54-408d-9371-b0f2267b3908","Type":"ContainerDied","Data":"85c9eef9762ccaef871c4918bfcdde2e328f1f6a0618caf722be508900a6a640"} Nov 01 23:06:47.248046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.247658 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85c9eef9762ccaef871c4918bfcdde2e328f1f6a0618caf722be508900a6a640" Nov 01 23:06:47.248046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.247553 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5" Nov 01 23:06:47.249910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.249862 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" event={"ID":"9a8ede17-ef50-4622-8ece-b80e35712dfd","Type":"ContainerStarted","Data":"cb034bb4bb4f96d0ef00446f9288f92dad7e04892343ebd3c1613cf6a090fa0c"} Nov 01 23:06:47.250079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.249936 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" event={"ID":"9a8ede17-ef50-4622-8ece-b80e35712dfd","Type":"ContainerStarted","Data":"ee47fc69e0f44bd901137d60e428f28063846a4468e70e6f8022e44e80485dc2"} Nov 01 23:06:47.250079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.249978 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:47.272745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.272696 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" podStartSLOduration=2.272682357 podStartE2EDuration="2.272682357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:47.268845054 +0000 UTC m=+2572.696722988" watchObservedRunningTime="2026-11-01 23:06:47.272682357 +0000 UTC m=+2572.700560288" Nov 01 23:06:47.530680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:47.530596 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:48.275541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:48.275509 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:48.982024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:48.981994 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:49.569730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:49.569694 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:49.788186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:49.788154 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:50.229603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.203109 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:50.597751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.597659 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:50.892988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.892908 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:50.915301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.915266 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb"] Nov 01 23:06:50.915602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.915555 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" podUID="ded9f23f-6082-4430-98c9-22b10e78ae0c" containerName="payload-processing" containerID="cri-o://48f7dbcef14d41a0ce215e8536c59b51c7a9bf6a3702105790bc7322cacf53a4" gracePeriod=30 Nov 01 23:06:50.932913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.932224 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2"] Nov 01 23:06:50.932913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.932446 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" podUID="ff5a8d8d-99bb-4706-9481-0320e3ff280f" containerName="payload-pre-processing" containerID="cri-o://a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957" gracePeriod=30 Nov 01 23:06:50.996378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:50.996347 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:51.043305 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:06:51.043271 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5a8d8d_99bb_4706_9481_0320e3ff280f.slice/crio-conmon-a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5a8d8d_99bb_4706_9481_0320e3ff280f.slice/crio-a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:06:51.043645 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:06:51.043611 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5a8d8d_99bb_4706_9481_0320e3ff280f.slice/crio-conmon-a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:06:51.134609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.132485 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:51.227597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.227085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:51.398645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.388685 4981 generic.go:356] "Generic (PLEG): container finished" podID="ff5a8d8d-99bb-4706-9481-0320e3ff280f" containerID="a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957" exitCode=0 Nov 01 23:06:51.398645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.388761 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" event={"ID":"ff5a8d8d-99bb-4706-9481-0320e3ff280f","Type":"ContainerDied","Data":"a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957"} Nov 01 23:06:51.398645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.394264 4981 generic.go:356] "Generic (PLEG): container finished" podID="ded9f23f-6082-4430-98c9-22b10e78ae0c" containerID="48f7dbcef14d41a0ce215e8536c59b51c7a9bf6a3702105790bc7322cacf53a4" exitCode=0 Nov 01 23:06:51.398645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.394494 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" event={"ID":"ded9f23f-6082-4430-98c9-22b10e78ae0c","Type":"ContainerDied","Data":"48f7dbcef14d41a0ce215e8536c59b51c7a9bf6a3702105790bc7322cacf53a4"} Nov 01 23:06:51.448674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.448644 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz"] Nov 01 23:06:51.454541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.454520 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fb5b4-rcgdz"] Nov 01 23:06:51.539647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.537066 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:06:51.539647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.539088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:51.658630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.658596 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:06:51.699601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.698766 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded9f23f-6082-4430-98c9-22b10e78ae0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded9f23f-6082-4430-98c9-22b10e78ae0c-plugins-config-volume\") pod \"ded9f23f-6082-4430-98c9-22b10e78ae0c\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " Nov 01 23:06:51.699601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.698981 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded9f23f-6082-4430-98c9-22b10e78ae0c-kube-api-access-pv7zn\" (UniqueName: \"kubernetes.io/projected/ded9f23f-6082-4430-98c9-22b10e78ae0c-kube-api-access-pv7zn\") pod \"ded9f23f-6082-4430-98c9-22b10e78ae0c\" (UID: \"ded9f23f-6082-4430-98c9-22b10e78ae0c\") " Nov 01 23:06:51.700906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.700873 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded9f23f-6082-4430-98c9-22b10e78ae0c-plugins-config-volume" pod "ded9f23f-6082-4430-98c9-22b10e78ae0c" (UID: "ded9f23f-6082-4430-98c9-22b10e78ae0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:51.702012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.701981 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded9f23f-6082-4430-98c9-22b10e78ae0c-kube-api-access-pv7zn" pod "ded9f23f-6082-4430-98c9-22b10e78ae0c" (UID: "ded9f23f-6082-4430-98c9-22b10e78ae0c"). InnerVolumeSpecName "kube-api-access-pv7zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:51.800867 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.800772 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5a8d8d-99bb-4706-9481-0320e3ff280f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5a8d8d-99bb-4706-9481-0320e3ff280f-plugins-config-volume\") pod \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " Nov 01 23:06:51.801050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.800917 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5a8d8d-99bb-4706-9481-0320e3ff280f-kube-api-access-svl22\" (UniqueName: \"kubernetes.io/projected/ff5a8d8d-99bb-4706-9481-0320e3ff280f-kube-api-access-svl22\") pod \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\" (UID: \"ff5a8d8d-99bb-4706-9481-0320e3ff280f\") " Nov 01 23:06:51.801143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.801119 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5a8d8d-99bb-4706-9481-0320e3ff280f-plugins-config-volume" pod "ff5a8d8d-99bb-4706-9481-0320e3ff280f" (UID: "ff5a8d8d-99bb-4706-9481-0320e3ff280f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:51.801683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.801658 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv7zn\" (UniqueName: \"kubernetes.io/projected/ded9f23f-6082-4430-98c9-22b10e78ae0c-kube-api-access-pv7zn\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:51.801812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.801702 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff5a8d8d-99bb-4706-9481-0320e3ff280f-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:51.801812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.801769 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded9f23f-6082-4430-98c9-22b10e78ae0c-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:51.803050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.803033 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5a8d8d-99bb-4706-9481-0320e3ff280f-kube-api-access-svl22" pod "ff5a8d8d-99bb-4706-9481-0320e3ff280f" (UID: "ff5a8d8d-99bb-4706-9481-0320e3ff280f"). InnerVolumeSpecName "kube-api-access-svl22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:51.897742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.897713 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:51.904111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:51.904085 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svl22\" (UniqueName: \"kubernetes.io/projected/ff5a8d8d-99bb-4706-9481-0320e3ff280f-kube-api-access-svl22\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:52.404490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.404455 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" Nov 01 23:06:52.404729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.404452 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2" event={"ID":"ff5a8d8d-99bb-4706-9481-0320e3ff280f","Type":"ContainerDied","Data":"2c168489f88173e803e61f82107f46a0c52fbdfbc3198cffd743f1f8a2ff09a3"} Nov 01 23:06:52.404729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.404621 4981 scope.go:122] "RemoveContainer" containerID="a1612de2303c2e3ef36a19897e62135fb335d46e9f461dc2352b9348cd794957" Nov 01 23:06:52.407226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.407203 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" Nov 01 23:06:52.407639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.407599 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb" event={"ID":"ded9f23f-6082-4430-98c9-22b10e78ae0c","Type":"ContainerDied","Data":"559f3c66e4d1569b9d77839ae1184d832a968da5aef41685f73f88eec7f89327"} Nov 01 23:06:52.420002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.419986 4981 scope.go:122] "RemoveContainer" containerID="48f7dbcef14d41a0ce215e8536c59b51c7a9bf6a3702105790bc7322cacf53a4" Nov 01 23:06:52.433806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.433783 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2"] Nov 01 23:06:52.471628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.458411 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fb5b4-c4478947-4n2z2"] Nov 01 23:06:52.475596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.471851 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb"] Nov 01 23:06:52.581459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.581428 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fb5b4-54747dbbfd-76swb"] Nov 01 23:06:52.738541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.738509 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6743b759-2915-4f26-8f10-d61662bb2a8a" path="/var/lib/kubelet/pods/6743b759-2915-4f26-8f10-d61662bb2a8a/volumes" Nov 01 23:06:52.739013 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.738998 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded9f23f-6082-4430-98c9-22b10e78ae0c" path="/var/lib/kubelet/pods/ded9f23f-6082-4430-98c9-22b10e78ae0c/volumes" Nov 01 23:06:52.739428 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.739414 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff5a8d8d-99bb-4706-9481-0320e3ff280f" path="/var/lib/kubelet/pods/ff5a8d8d-99bb-4706-9481-0320e3ff280f/volumes" Nov 01 23:06:52.977392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:52.977365 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:53.766094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.766059 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz"] Nov 01 23:06:53.766959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.766929 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5a8d29e3-8c54-408d-9371-b0f2267b3908" containerName="revoke-keys" Nov 01 23:06:53.766959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.766948 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a8d29e3-8c54-408d-9371-b0f2267b3908" containerName="revoke-keys" Nov 01 23:06:53.767170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.766972 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ff5a8d8d-99bb-4706-9481-0320e3ff280f" containerName="payload-pre-processing" Nov 01 23:06:53.767170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.766981 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5a8d8d-99bb-4706-9481-0320e3ff280f" containerName="payload-pre-processing" Nov 01 23:06:53.767170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.767011 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ded9f23f-6082-4430-98c9-22b10e78ae0c" containerName="payload-processing" Nov 01 23:06:53.767170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.767017 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded9f23f-6082-4430-98c9-22b10e78ae0c" containerName="payload-processing" Nov 01 23:06:53.767170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.767166 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a8d29e3-8c54-408d-9371-b0f2267b3908" containerName="revoke-keys" Nov 01 23:06:53.767452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.767177 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5a8d8d-99bb-4706-9481-0320e3ff280f" containerName="payload-pre-processing" Nov 01 23:06:53.767452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.767186 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded9f23f-6082-4430-98c9-22b10e78ae0c" containerName="payload-processing" Nov 01 23:06:53.774745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.774720 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.776866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.776843 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz"] Nov 01 23:06:53.781658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.781636 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:06:53.784340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.784020 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:06:53.859368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.859325 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.859526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.859493 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.859815 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.859793 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d965\" (UniqueName: \"kubernetes.io/projected/3f40cf22-b453-4c6e-89e9-f08d11c413d9-kube-api-access-4d965\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.948784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.948755 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:53.961845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.961819 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.962018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.961933 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.962159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.962143 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d965\" (UniqueName: \"kubernetes.io/projected/3f40cf22-b453-4c6e-89e9-f08d11c413d9-kube-api-access-4d965\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.962540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.962516 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.964333 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.964310 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:53.970108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:53.970085 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d965\" (UniqueName: \"kubernetes.io/projected/3f40cf22-b453-4c6e-89e9-f08d11c413d9-kube-api-access-4d965\") pod \"maas-api-revoke-sub-c6fe131d-wmjwz\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:54.083352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:54.083248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:54.091761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:54.091736 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:54.181062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:54.180493 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:54.378244 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:06:54.378204 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f40cf22_b453_4c6e_89e9_f08d11c413d9.slice/crio-9a838eaae8f9030d9db2dedbf93e3441073b4306ce8e2fb6e45e562dcdea5f5d WatchSource:0}: Error finding container 9a838eaae8f9030d9db2dedbf93e3441073b4306ce8e2fb6e45e562dcdea5f5d: Status 404 returned error can't find the container with id 9a838eaae8f9030d9db2dedbf93e3441073b4306ce8e2fb6e45e562dcdea5f5d Nov 01 23:06:54.378794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:54.378769 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz"] Nov 01 23:06:54.436139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:54.436111 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" event={"ID":"3f40cf22-b453-4c6e-89e9-f08d11c413d9","Type":"ContainerStarted","Data":"9a838eaae8f9030d9db2dedbf93e3441073b4306ce8e2fb6e45e562dcdea5f5d"} Nov 01 23:06:54.887712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:54.887683 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:55.337331 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:55.337195 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:55.446319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:55.446289 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:55.447236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:55.447209 4981 generic.go:356] "Generic (PLEG): container finished" podID="3f40cf22-b453-4c6e-89e9-f08d11c413d9" containerID="ae4aacdda4ea3a9b5e0b5ca655e8dfcbf93ffd966a484b4d5cffec0853896a04" exitCode=0 Nov 01 23:06:55.447387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:55.447292 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" event={"ID":"3f40cf22-b453-4c6e-89e9-f08d11c413d9","Type":"ContainerDied","Data":"ae4aacdda4ea3a9b5e0b5ca655e8dfcbf93ffd966a484b4d5cffec0853896a04"} Nov 01 23:06:55.868534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:55.868449 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:56.593793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.593754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:56.634554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.634209 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:56.817262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.817182 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-xks-ca\") pod \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " Nov 01 23:06:56.817423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.817334 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f40cf22-b453-4c6e-89e9-f08d11c413d9-kube-api-access-4d965\" (UniqueName: \"kubernetes.io/projected/3f40cf22-b453-4c6e-89e9-f08d11c413d9-kube-api-access-4d965\") pod \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " Nov 01 23:06:56.817478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.817438 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-openshift-ca\") pod \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\" (UID: \"3f40cf22-b453-4c6e-89e9-f08d11c413d9\") " Nov 01 23:06:56.817892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.817862 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-openshift-ca" pod "3f40cf22-b453-4c6e-89e9-f08d11c413d9" (UID: "3f40cf22-b453-4c6e-89e9-f08d11c413d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:06:56.818170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.818148 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:56.819433 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.819410 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f40cf22-b453-4c6e-89e9-f08d11c413d9-kube-api-access-4d965" pod "3f40cf22-b453-4c6e-89e9-f08d11c413d9" (UID: "3f40cf22-b453-4c6e-89e9-f08d11c413d9"). InnerVolumeSpecName "kube-api-access-4d965". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:06:56.819516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.819460 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-xks-ca" pod "3f40cf22-b453-4c6e-89e9-f08d11c413d9" (UID: "3f40cf22-b453-4c6e-89e9-f08d11c413d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:06:56.920360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.920320 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f40cf22-b453-4c6e-89e9-f08d11c413d9-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:56.920360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:56.920358 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d965\" (UniqueName: \"kubernetes.io/projected/3f40cf22-b453-4c6e-89e9-f08d11c413d9-kube-api-access-4d965\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:06:57.253734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:57.253704 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:06:57.354773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:57.354732 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:57.490339 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:57.490303 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" Nov 01 23:06:57.490516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:57.490301 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz" event={"ID":"3f40cf22-b453-4c6e-89e9-f08d11c413d9","Type":"ContainerDied","Data":"9a838eaae8f9030d9db2dedbf93e3441073b4306ce8e2fb6e45e562dcdea5f5d"} Nov 01 23:06:57.490516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:57.490437 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a838eaae8f9030d9db2dedbf93e3441073b4306ce8e2fb6e45e562dcdea5f5d" Nov 01 23:06:58.097165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:58.097103 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:58.265231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:58.265199 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:06:58.613962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:58.613926 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:58.726899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:58.726865 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:58.899351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:58.899267 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7"] Nov 01 23:06:58.899535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:58.899474 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" containerID="cri-o://55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6" gracePeriod=30 Nov 01 23:06:58.936562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:58.936528 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:59.025766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.025730 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:06:59.381325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.381289 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv"] Nov 01 23:06:59.382188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.382170 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3f40cf22-b453-4c6e-89e9-f08d11c413d9" containerName="revoke-keys" Nov 01 23:06:59.382188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.382189 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f40cf22-b453-4c6e-89e9-f08d11c413d9" containerName="revoke-keys" Nov 01 23:06:59.382329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.382320 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f40cf22-b453-4c6e-89e9-f08d11c413d9" containerName="revoke-keys" Nov 01 23:06:59.386423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.386405 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.395308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.395279 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv"] Nov 01 23:06:59.485763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.485732 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:06:59.485929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.485776 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:06:59.515237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.515210 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.515378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.515260 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.515423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.515395 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sknh\" (UniqueName: \"kubernetes.io/projected/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-kube-api-access-7sknh\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.617168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.617137 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.617340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.617196 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.617340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.617246 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sknh\" (UniqueName: \"kubernetes.io/projected/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-kube-api-access-7sknh\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.617920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.617897 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.619511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.619486 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.627502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.627482 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sknh\" (UniqueName: \"kubernetes.io/projected/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-kube-api-access-7sknh\") pod \"maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.714773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.714743 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:06:59.945440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:06:59.945404 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:00.034026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:00.033922 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv"] Nov 01 23:07:00.037929 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:00.037896 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc09190f_5adb_4d4e_9e9b_4a24c7dbd313.slice/crio-420cdd9df33b8531778196959eaef12ad5bbb01e6d2bf820b490efcb05209769 WatchSource:0}: Error finding container 420cdd9df33b8531778196959eaef12ad5bbb01e6d2bf820b490efcb05209769: Status 404 returned error can't find the container with id 420cdd9df33b8531778196959eaef12ad5bbb01e6d2bf820b490efcb05209769 Nov 01 23:07:00.517126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:00.517090 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" event={"ID":"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313","Type":"ContainerStarted","Data":"420cdd9df33b8531778196959eaef12ad5bbb01e6d2bf820b490efcb05209769"} Nov 01 23:07:00.652644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:00.652540 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:01.333362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:01.333328 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:01.527091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:01.527057 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" containerID="fe48403743142f1823ea250d53c85ff4eed58e3fa0372302281e694f6bbafb1c" exitCode=0 Nov 01 23:07:01.527503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:01.527139 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" event={"ID":"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313","Type":"ContainerDied","Data":"fe48403743142f1823ea250d53c85ff4eed58e3fa0372302281e694f6bbafb1c"} Nov 01 23:07:02.141089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.141040 4981 scope.go:122] "RemoveContainer" containerID="10c5aeb43fa027dc0eb207262156ece9794ba17b9fecc394cb8ac7fa90d5140b" Nov 01 23:07:02.156160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.156131 4981 scope.go:122] "RemoveContainer" containerID="4da3f1f9163f832a42ed5d522989b33c0a9f690a96fd425070a99a16cb9359e3" Nov 01 23:07:02.168155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.168127 4981 scope.go:122] "RemoveContainer" containerID="233ab7b80b03339f629a8cfcf9d336f257e90d8dfa06ff73173661ef874460c6" Nov 01 23:07:02.178697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.178669 4981 scope.go:122] "RemoveContainer" containerID="1e39ef2c93737dd469452bcacbfa4eeb203f428ee3d65a9789adea1cffcd6dc3" Nov 01 23:07:02.181275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.181250 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:02.642167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.642132 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4"] Nov 01 23:07:02.652663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.652559 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.653782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.653721 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4"] Nov 01 23:07:02.746170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.746144 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:07:02.787963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.787880 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.788125 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.788028 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.788314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.788288 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbhh\" (UniqueName: \"kubernetes.io/projected/90d9749b-e6a8-433b-8697-67d2a98ae8d0-kube-api-access-kdbhh\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.890037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.890000 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-kube-api-access-7sknh\" (UniqueName: \"kubernetes.io/projected/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-kube-api-access-7sknh\") pod \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " Nov 01 23:07:02.890247 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.890183 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-xks-ca\") pod \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " Nov 01 23:07:02.890439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.890424 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-openshift-ca\") pod \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\" (UID: \"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313\") " Nov 01 23:07:02.890816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.890786 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-openshift-ca" pod "dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" (UID: "dc09190f-5adb-4d4e-9e9b-4a24c7dbd313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:02.890974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.890789 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbhh\" (UniqueName: \"kubernetes.io/projected/90d9749b-e6a8-433b-8697-67d2a98ae8d0-kube-api-access-kdbhh\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.891062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.891043 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.891164 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.891151 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.891465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.891446 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:02.891886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.891861 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.892408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.892384 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-kube-api-access-7sknh" pod "dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" (UID: "dc09190f-5adb-4d4e-9e9b-4a24c7dbd313"). InnerVolumeSpecName "kube-api-access-7sknh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:02.892527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.892511 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-xks-ca" pod "dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" (UID: "dc09190f-5adb-4d4e-9e9b-4a24c7dbd313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:02.893316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.893297 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.899541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.899519 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbhh\" (UniqueName: \"kubernetes.io/projected/90d9749b-e6a8-433b-8697-67d2a98ae8d0-kube-api-access-kdbhh\") pod \"maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.990089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.990051 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:02.992526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.992496 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sknh\" (UniqueName: \"kubernetes.io/projected/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-kube-api-access-7sknh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:02.992526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:02.992525 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:03.003361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.003323 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:07:03.003535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.003382 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:07:03.260062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.260030 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:03.281015 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:03.280981 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d9749b_e6a8_433b_8697_67d2a98ae8d0.slice/crio-6d82dd5aa5f7c5b145821d49c034177eaaa360270094f9091b28980d75dda4d7 WatchSource:0}: Error finding container 6d82dd5aa5f7c5b145821d49c034177eaaa360270094f9091b28980d75dda4d7: Status 404 returned error can't find the container with id 6d82dd5aa5f7c5b145821d49c034177eaaa360270094f9091b28980d75dda4d7 Nov 01 23:07:03.281604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.281541 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4"] Nov 01 23:07:03.545771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.545737 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" Nov 01 23:07:03.545981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.545773 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv" event={"ID":"dc09190f-5adb-4d4e-9e9b-4a24c7dbd313","Type":"ContainerDied","Data":"420cdd9df33b8531778196959eaef12ad5bbb01e6d2bf820b490efcb05209769"} Nov 01 23:07:03.545981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.545826 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="420cdd9df33b8531778196959eaef12ad5bbb01e6d2bf820b490efcb05209769" Nov 01 23:07:03.547370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.547342 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" event={"ID":"90d9749b-e6a8-433b-8697-67d2a98ae8d0","Type":"ContainerStarted","Data":"6d82dd5aa5f7c5b145821d49c034177eaaa360270094f9091b28980d75dda4d7"} Nov 01 23:07:03.953161 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:03.953093 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:04.241971 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.241948 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:07:04.316503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316474 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-podinfo\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.316695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316624 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16cbedbb-ef98-4fc8-927b-a47634d5e428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cbedbb-ef98-4fc8-927b-a47634d5e428-istiod-ca-cert\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.316762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316740 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-credential-socket\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.316828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316815 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-data\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.316889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316872 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-certs\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.316943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316888 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-socket\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.316943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316917 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-envoy\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.317050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.316942 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-kube-api-access-qztt9\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-kube-api-access-qztt9\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.317050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.317017 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-token\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-token\") pod \"16cbedbb-ef98-4fc8-927b-a47634d5e428\" (UID: \"16cbedbb-ef98-4fc8-927b-a47634d5e428\") " Nov 01 23:07:04.317813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.317415 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-socket" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:04.317813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.317540 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16cbedbb-ef98-4fc8-927b-a47634d5e428-istiod-ca-cert" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:04.317813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.317777 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cbedbb-ef98-4fc8-927b-a47634d5e428-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.317813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.317801 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.318257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.317989 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-credential-socket" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:04.318257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.318029 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-certs" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:04.318691 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.318667 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-data" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:04.323629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.321074 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-kube-api-access-qztt9" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "kube-api-access-qztt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:04.323629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.321778 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-envoy" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:04.323629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.322972 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-token" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:04.324798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.324772 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-podinfo" pod "16cbedbb-ef98-4fc8-927b-a47634d5e428" (UID: "16cbedbb-ef98-4fc8-927b-a47634d5e428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:07:04.385976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.385949 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn"] Nov 01 23:07:04.386817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.386796 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" containerName="revoke-keys" Nov 01 23:07:04.386817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.386813 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" containerName="revoke-keys" Nov 01 23:07:04.386941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.386836 4981 cpu_manager.go:414] "containerMap: removing container" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" Nov 01 23:07:04.386941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.386842 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" Nov 01 23:07:04.386941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.386874 4981 cpu_manager.go:414] "containerMap: removing container" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" Nov 01 23:07:04.386941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.386880 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" Nov 01 23:07:04.387091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.387000 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" Nov 01 23:07:04.387091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.387014 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerName="istio-proxy" Nov 01 23:07:04.387091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.387026 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" containerName="revoke-keys" Nov 01 23:07:04.391214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.391191 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.399784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.399758 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn"] Nov 01 23:07:04.420095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.420062 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.420095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.420097 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.420232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.420112 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qztt9\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-kube-api-access-qztt9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.420232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.420127 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.420232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.420140 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.420232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.420153 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.420232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.420166 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cbedbb-ef98-4fc8-927b-a47634d5e428-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:04.521641 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.521542 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qmv\" (UniqueName: \"kubernetes.io/projected/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-kube-api-access-v2qmv\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.521768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.521673 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.521768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.521734 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.557055 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.557024 4981 generic.go:356] "Generic (PLEG): container finished" podID="90d9749b-e6a8-433b-8697-67d2a98ae8d0" containerID="befe779c5945cd508efa960509fa8bdf163f3ad1eb6a7f5c664a5a004758e040" exitCode=0 Nov 01 23:07:04.557195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.557104 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" event={"ID":"90d9749b-e6a8-433b-8697-67d2a98ae8d0","Type":"ContainerDied","Data":"befe779c5945cd508efa960509fa8bdf163f3ad1eb6a7f5c664a5a004758e040"} Nov 01 23:07:04.558949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.558928 4981 generic.go:356] "Generic (PLEG): container finished" podID="16cbedbb-ef98-4fc8-927b-a47634d5e428" containerID="55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6" exitCode=0 Nov 01 23:07:04.559084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.558980 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" event={"ID":"16cbedbb-ef98-4fc8-927b-a47634d5e428","Type":"ContainerDied","Data":"55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6"} Nov 01 23:07:04.559084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.558988 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" Nov 01 23:07:04.559084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.559026 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7" event={"ID":"16cbedbb-ef98-4fc8-927b-a47634d5e428","Type":"ContainerDied","Data":"c3bf3876280f2a29ef1aa1c2bff11c9b34b70d92938f3026aadfb195a99fbd8c"} Nov 01 23:07:04.559084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.559055 4981 scope.go:122] "RemoveContainer" containerID="55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6" Nov 01 23:07:04.571493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.571477 4981 scope.go:122] "RemoveContainer" containerID="c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492" Nov 01 23:07:04.583626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.583609 4981 scope.go:122] "RemoveContainer" containerID="55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6" Nov 01 23:07:04.583891 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:04.583873 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6\": container with ID starting with 55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6 not found: ID does not exist" containerID="55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6" Nov 01 23:07:04.583950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.583901 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6"} err="failed to get container status \"55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6\": rpc error: code = NotFound desc = could not find container \"55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6\": container with ID starting with 55447e9259c3a95509d43b3dfe857ebb30fbf678542baa3fc780251e585449e6 not found: ID does not exist" Nov 01 23:07:04.583950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.583917 4981 scope.go:122] "RemoveContainer" containerID="c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492" Nov 01 23:07:04.584188 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:04.584170 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492\": container with ID starting with c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492 not found: ID does not exist" containerID="c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492" Nov 01 23:07:04.584246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.584196 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492"} err="failed to get container status \"c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492\": rpc error: code = NotFound desc = could not find container \"c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492\": container with ID starting with c910484fc435ad86773ca2005f15ca9bda84e28b474a28a1c82735b7f14cf492 not found: ID does not exist" Nov 01 23:07:04.594445 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.594422 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7"] Nov 01 23:07:04.598408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.598387 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fb5b4-openshift-default-5bb5b68655-srkh7"] Nov 01 23:07:04.622881 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.622859 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qmv\" (UniqueName: \"kubernetes.io/projected/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-kube-api-access-v2qmv\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.622964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.622890 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.623001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.622960 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.623541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.623515 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.625219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.625201 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.631405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.631382 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qmv\" (UniqueName: \"kubernetes.io/projected/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-kube-api-access-v2qmv\") pod \"maas-api-revoke-sub-2c322b61-kfgcn\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.709652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.709628 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:04.740828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.740798 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16cbedbb-ef98-4fc8-927b-a47634d5e428" path="/var/lib/kubelet/pods/16cbedbb-ef98-4fc8-927b-a47634d5e428/volumes" Nov 01 23:07:04.749400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:04.749371 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:04.999873 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:04.999833 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bed368_31e8_4ab9_aaee_cc8b501d0e9e.slice/crio-39bccad944b2dacd2efd626434adb2b85b005c4a8d142494f0422e38764e1ff2 WatchSource:0}: Error finding container 39bccad944b2dacd2efd626434adb2b85b005c4a8d142494f0422e38764e1ff2: Status 404 returned error can't find the container with id 39bccad944b2dacd2efd626434adb2b85b005c4a8d142494f0422e38764e1ff2 Nov 01 23:07:05.000307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.000181 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn"] Nov 01 23:07:05.457115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.457083 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:05.571901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.571865 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" event={"ID":"12bed368-31e8-4ab9-aaee-cc8b501d0e9e","Type":"ContainerStarted","Data":"2a977c6af08e52d2a37d553ac2c9adf5012b04a9af3038eca9744bcee6c4753a"} Nov 01 23:07:05.572095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.571929 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" event={"ID":"12bed368-31e8-4ab9-aaee-cc8b501d0e9e","Type":"ContainerStarted","Data":"39bccad944b2dacd2efd626434adb2b85b005c4a8d142494f0422e38764e1ff2"} Nov 01 23:07:05.595354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.595296 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" podStartSLOduration=1.595280436 podStartE2EDuration="1.595280436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:05.590315832 +0000 UTC m=+2591.018193763" watchObservedRunningTime="2026-11-01 23:07:05.595280436 +0000 UTC m=+2591.023158367" Nov 01 23:07:05.762564 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.762534 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:05.865259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.865218 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-xks-ca\") pod \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " Nov 01 23:07:05.865480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.865461 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-openshift-ca\") pod \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " Nov 01 23:07:05.865562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.865545 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d9749b-e6a8-433b-8697-67d2a98ae8d0-kube-api-access-kdbhh\" (UniqueName: \"kubernetes.io/projected/90d9749b-e6a8-433b-8697-67d2a98ae8d0-kube-api-access-kdbhh\") pod \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\" (UID: \"90d9749b-e6a8-433b-8697-67d2a98ae8d0\") " Nov 01 23:07:05.865836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.865813 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-openshift-ca" pod "90d9749b-e6a8-433b-8697-67d2a98ae8d0" (UID: "90d9749b-e6a8-433b-8697-67d2a98ae8d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:05.866156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.866139 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:05.867419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.867394 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-xks-ca" pod "90d9749b-e6a8-433b-8697-67d2a98ae8d0" (UID: "90d9749b-e6a8-433b-8697-67d2a98ae8d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:05.867749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.867727 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d9749b-e6a8-433b-8697-67d2a98ae8d0-kube-api-access-kdbhh" pod "90d9749b-e6a8-433b-8697-67d2a98ae8d0" (UID: "90d9749b-e6a8-433b-8697-67d2a98ae8d0"). InnerVolumeSpecName "kube-api-access-kdbhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:05.967137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.967108 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d9749b-e6a8-433b-8697-67d2a98ae8d0-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:05.967137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:05.967139 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbhh\" (UniqueName: \"kubernetes.io/projected/90d9749b-e6a8-433b-8697-67d2a98ae8d0-kube-api-access-kdbhh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:06.026143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.026108 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r"] Nov 01 23:07:06.030659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.030630 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd53567-qw92r"] Nov 01 23:07:06.162038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.162002 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:06.581448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.581409 4981 generic.go:356] "Generic (PLEG): container finished" podID="12bed368-31e8-4ab9-aaee-cc8b501d0e9e" containerID="2a977c6af08e52d2a37d553ac2c9adf5012b04a9af3038eca9744bcee6c4753a" exitCode=0 Nov 01 23:07:06.581664 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.581498 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" event={"ID":"12bed368-31e8-4ab9-aaee-cc8b501d0e9e","Type":"ContainerDied","Data":"2a977c6af08e52d2a37d553ac2c9adf5012b04a9af3038eca9744bcee6c4753a"} Nov 01 23:07:06.582890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.582871 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" Nov 01 23:07:06.582995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.582904 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4" event={"ID":"90d9749b-e6a8-433b-8697-67d2a98ae8d0","Type":"ContainerDied","Data":"6d82dd5aa5f7c5b145821d49c034177eaaa360270094f9091b28980d75dda4d7"} Nov 01 23:07:06.582995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.582947 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d82dd5aa5f7c5b145821d49c034177eaaa360270094f9091b28980d75dda4d7" Nov 01 23:07:06.737895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.737857 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db084668-3f72-4b0d-92b1-49e1b732e361" path="/var/lib/kubelet/pods/db084668-3f72-4b0d-92b1-49e1b732e361/volumes" Nov 01 23:07:06.887482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:06.887402 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:07.643228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.643196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:07.821650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.821618 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:07.953556 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.953523 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-kube-api-access-v2qmv\" (UniqueName: \"kubernetes.io/projected/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-kube-api-access-v2qmv\") pod \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " Nov 01 23:07:07.953782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.953671 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-openshift-ca\") pod \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " Nov 01 23:07:07.953891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.953872 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-xks-ca\") pod \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\" (UID: \"12bed368-31e8-4ab9-aaee-cc8b501d0e9e\") " Nov 01 23:07:07.954082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.954060 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-openshift-ca" pod "12bed368-31e8-4ab9-aaee-cc8b501d0e9e" (UID: "12bed368-31e8-4ab9-aaee-cc8b501d0e9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:07.954477 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.954460 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:07.955731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.955706 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-kube-api-access-v2qmv" pod "12bed368-31e8-4ab9-aaee-cc8b501d0e9e" (UID: "12bed368-31e8-4ab9-aaee-cc8b501d0e9e"). InnerVolumeSpecName "kube-api-access-v2qmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:07.955935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:07.955920 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-xks-ca" pod "12bed368-31e8-4ab9-aaee-cc8b501d0e9e" (UID: "12bed368-31e8-4ab9-aaee-cc8b501d0e9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:08.056739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.056700 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:08.056739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.056735 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2qmv\" (UniqueName: \"kubernetes.io/projected/12bed368-31e8-4ab9-aaee-cc8b501d0e9e-kube-api-access-v2qmv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:08.384250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.384176 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l"] Nov 01 23:07:08.385033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.384974 4981 cpu_manager.go:414] "containerMap: removing container" podUID="12bed368-31e8-4ab9-aaee-cc8b501d0e9e" containerName="revoke-keys" Nov 01 23:07:08.385033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.384990 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bed368-31e8-4ab9-aaee-cc8b501d0e9e" containerName="revoke-keys" Nov 01 23:07:08.385179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.385102 4981 cpu_manager.go:414] "containerMap: removing container" podUID="90d9749b-e6a8-433b-8697-67d2a98ae8d0" containerName="revoke-keys" Nov 01 23:07:08.385179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.385109 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d9749b-e6a8-433b-8697-67d2a98ae8d0" containerName="revoke-keys" Nov 01 23:07:08.385300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.385251 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d9749b-e6a8-433b-8697-67d2a98ae8d0" containerName="revoke-keys" Nov 01 23:07:08.385300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.385265 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bed368-31e8-4ab9-aaee-cc8b501d0e9e" containerName="revoke-keys" Nov 01 23:07:08.388841 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.388823 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.393585 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.393528 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a9f94e-openshift-default-dockercfg-t57nk\"" Nov 01 23:07:08.401455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.401432 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l"] Nov 01 23:07:08.461722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.461686 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.461897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.461872 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-envoy\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.461946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.461908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-data\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.462012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.462000 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wdw\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-kube-api-access-b6wdw\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.462081 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.462068 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-certs\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.462120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.462103 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-credential-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.462168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.462122 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-token\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.462221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.462202 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-podinfo\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.462285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.462273 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istiod-ca-cert\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.503868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.503838 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.575700 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.575895 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-envoy\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.575928 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-data\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.576002 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wdw\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-kube-api-access-b6wdw\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.576091 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-certs\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.576127 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-credential-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.576151 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-token\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.576221 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-podinfo\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.581601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.576261 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istiod-ca-cert\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.586898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.586862 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.587973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.587356 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-certs\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.588199 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.587633 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-credential-socket\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.588331 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.587897 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-data\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.588497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.588472 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istiod-ca-cert\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.599766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.596915 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-envoy\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.599766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.598702 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-podinfo\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.600799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.600771 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-token\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.600918 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.600903 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wdw\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-kube-api-access-b6wdw\") pod \"e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.608089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.608066 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" Nov 01 23:07:08.608183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.608090 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn" event={"ID":"12bed368-31e8-4ab9-aaee-cc8b501d0e9e","Type":"ContainerDied","Data":"39bccad944b2dacd2efd626434adb2b85b005c4a8d142494f0422e38764e1ff2"} Nov 01 23:07:08.608183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.608150 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39bccad944b2dacd2efd626434adb2b85b005c4a8d142494f0422e38764e1ff2" Nov 01 23:07:08.707034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.707005 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:08.997701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:08.997670 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l"] Nov 01 23:07:08.997902 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:08.997746 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c914dd3_3527_40a0_8dfc_a2a884e0f49b.slice/crio-dd1e5c228188e309857eb9302a1ad4eb2e28a9a101c7b879b978a1a09257c1fd WatchSource:0}: Error finding container dd1e5c228188e309857eb9302a1ad4eb2e28a9a101c7b879b978a1a09257c1fd: Status 404 returned error can't find the container with id dd1e5c228188e309857eb9302a1ad4eb2e28a9a101c7b879b978a1a09257c1fd Nov 01 23:07:09.001179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.001141 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:07:09.001287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.001246 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:07:09.001334 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.001293 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:07:09.080964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.080937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:09.618529 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.618494 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" event={"ID":"7c914dd3-3527-40a0-8dfc-a2a884e0f49b","Type":"ContainerStarted","Data":"a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e"} Nov 01 23:07:09.618769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.618546 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" event={"ID":"7c914dd3-3527-40a0-8dfc-a2a884e0f49b","Type":"ContainerStarted","Data":"dd1e5c228188e309857eb9302a1ad4eb2e28a9a101c7b879b978a1a09257c1fd"} Nov 01 23:07:09.640233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.640189 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" podStartSLOduration=1.640176189 podStartE2EDuration="1.640176189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:09.638000262 +0000 UTC m=+2595.065878193" watchObservedRunningTime="2026-11-01 23:07:09.640176189 +0000 UTC m=+2595.068054120" Nov 01 23:07:09.707123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.707089 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:09.796600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:09.773794 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:10.669103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:10.669076 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:10.709551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:10.709519 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" start-of-body= Nov 01 23:07:10.710008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:10.709570 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" Nov 01 23:07:11.287611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:11.287000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:11.708400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:11.708360 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" start-of-body= Nov 01 23:07:11.708592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:11.708441 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" Nov 01 23:07:11.952496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:11.952462 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:12.603357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.603321 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:12.634099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.634065 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c"] Nov 01 23:07:12.634449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.634422 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" podUID="8e865675-b66f-4bc2-af3d-98477f386c8e" containerName="payload-processing" containerID="cri-o://bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66" gracePeriod=30 Nov 01 23:07:12.641708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.641685 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28"] Nov 01 23:07:12.641899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.641879 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" podUID="52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" containerName="payload-pre-processing" containerID="cri-o://3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a" gracePeriod=30 Nov 01 23:07:12.721302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.719515 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:12.721302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.719658 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" start-of-body= Nov 01 23:07:12.721302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.719713 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" Nov 01 23:07:12.851755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.851675 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:12.975313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:12.975113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:13.121693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.121627 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv"] Nov 01 23:07:13.131984 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.131935 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-85b1ad-j66bv"] Nov 01 23:07:13.234680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.234544 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:13.426750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.419956 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:13.458669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.458286 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:07:13.508601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.508496 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:13.528323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.528299 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:07:13.530202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.530177 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-kube-api-access-kr429\" (UniqueName: \"kubernetes.io/projected/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-kube-api-access-kr429\") pod \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " Nov 01 23:07:13.530322 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.530261 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-plugins-config-volume\") pod \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\" (UID: \"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0\") " Nov 01 23:07:13.530876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.530854 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-plugins-config-volume" pod "52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" (UID: "52a03d5b-6491-4f44-9ea6-dc6f1e612ca0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:13.531129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.531110 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:13.532978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.532955 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-kube-api-access-kr429" pod "52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" (UID: "52a03d5b-6491-4f44-9ea6-dc6f1e612ca0"). InnerVolumeSpecName "kube-api-access-kr429". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:13.632702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.632668 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e865675-b66f-4bc2-af3d-98477f386c8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865675-b66f-4bc2-af3d-98477f386c8e-plugins-config-volume\") pod \"8e865675-b66f-4bc2-af3d-98477f386c8e\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " Nov 01 23:07:13.633116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.633084 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e865675-b66f-4bc2-af3d-98477f386c8e-kube-api-access-qqsp8\" (UniqueName: \"kubernetes.io/projected/8e865675-b66f-4bc2-af3d-98477f386c8e-kube-api-access-qqsp8\") pod \"8e865675-b66f-4bc2-af3d-98477f386c8e\" (UID: \"8e865675-b66f-4bc2-af3d-98477f386c8e\") " Nov 01 23:07:13.633361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.633320 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e865675-b66f-4bc2-af3d-98477f386c8e-plugins-config-volume" pod "8e865675-b66f-4bc2-af3d-98477f386c8e" (UID: "8e865675-b66f-4bc2-af3d-98477f386c8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:13.633840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.633818 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr429\" (UniqueName: \"kubernetes.io/projected/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0-kube-api-access-kr429\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:13.633951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.633844 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865675-b66f-4bc2-af3d-98477f386c8e-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:13.635234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.635207 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e865675-b66f-4bc2-af3d-98477f386c8e-kube-api-access-qqsp8" pod "8e865675-b66f-4bc2-af3d-98477f386c8e" (UID: "8e865675-b66f-4bc2-af3d-98477f386c8e"). InnerVolumeSpecName "kube-api-access-qqsp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:13.672768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.672742 4981 generic.go:356] "Generic (PLEG): container finished" podID="52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" containerID="3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a" exitCode=0 Nov 01 23:07:13.672875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.672793 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" Nov 01 23:07:13.672875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.672828 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" event={"ID":"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0","Type":"ContainerDied","Data":"3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a"} Nov 01 23:07:13.672993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.672885 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28" event={"ID":"52a03d5b-6491-4f44-9ea6-dc6f1e612ca0","Type":"ContainerDied","Data":"b4f0a5599c5dd53e08729038f35ce45f659a5423517268c10f762e2736e56718"} Nov 01 23:07:13.672993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.672909 4981 scope.go:122] "RemoveContainer" containerID="3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a" Nov 01 23:07:13.701362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.701330 4981 generic.go:356] "Generic (PLEG): container finished" podID="8e865675-b66f-4bc2-af3d-98477f386c8e" containerID="bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66" exitCode=0 Nov 01 23:07:13.701490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.701382 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" Nov 01 23:07:13.701555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.701540 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" event={"ID":"8e865675-b66f-4bc2-af3d-98477f386c8e","Type":"ContainerDied","Data":"bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66"} Nov 01 23:07:13.701638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.701611 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c" event={"ID":"8e865675-b66f-4bc2-af3d-98477f386c8e","Type":"ContainerDied","Data":"b4d85da1dac8d358154092a47b4d8fe51bda085f7e9e8edb44b47410568ed82d"} Nov 01 23:07:13.707825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.707354 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" start-of-body= Nov 01 23:07:13.707825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.707407 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" Nov 01 23:07:13.712862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.712840 4981 scope.go:122] "RemoveContainer" containerID="3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a" Nov 01 23:07:13.713173 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:13.713151 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a\": container with ID starting with 3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a not found: ID does not exist" containerID="3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a" Nov 01 23:07:13.713238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.713186 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a"} err="failed to get container status \"3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a\": rpc error: code = NotFound desc = could not find container \"3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a\": container with ID starting with 3ec70895bd4de7431e35558b4822f630deb7ade4d1ec3c4b689f614612ff8f4a not found: ID does not exist" Nov 01 23:07:13.713238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.713211 4981 scope.go:122] "RemoveContainer" containerID="bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66" Nov 01 23:07:13.724790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.724773 4981 scope.go:122] "RemoveContainer" containerID="bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66" Nov 01 23:07:13.725050 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:13.725030 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66\": container with ID starting with bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66 not found: ID does not exist" containerID="bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66" Nov 01 23:07:13.725131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.725062 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66"} err="failed to get container status \"bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66\": rpc error: code = NotFound desc = could not find container \"bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66\": container with ID starting with bb8d874f5ee9b5a717f09929c219710be5090f455b973201349df442df8f0b66 not found: ID does not exist" Nov 01 23:07:13.726003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.725984 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28"] Nov 01 23:07:13.733502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.733477 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-85b1ad-645bbd78ff-64m28"] Nov 01 23:07:13.735123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.735098 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqsp8\" (UniqueName: \"kubernetes.io/projected/8e865675-b66f-4bc2-af3d-98477f386c8e-kube-api-access-qqsp8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:13.746861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.746840 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c"] Nov 01 23:07:13.751263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:13.751238 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-85b1ad-78989ff8cd-ttk8c"] Nov 01 23:07:14.346207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.346175 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:14.527559 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:14.527521 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:07:14.707284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.707244 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" start-of-body= Nov 01 23:07:14.707492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.707299 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.68:15021/healthz/ready\": dial tcp 10.134.1.68:15021: connect: connection refused" Nov 01 23:07:14.736316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.736283 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" path="/var/lib/kubelet/pods/52a03d5b-6491-4f44-9ea6-dc6f1e612ca0/volumes" Nov 01 23:07:14.736970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.736942 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e865675-b66f-4bc2-af3d-98477f386c8e" path="/var/lib/kubelet/pods/8e865675-b66f-4bc2-af3d-98477f386c8e/volumes" Nov 01 23:07:14.737534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.737517 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc09190f-5adb-4d4e-9e9b-4a24c7dbd313" path="/var/lib/kubelet/pods/dc09190f-5adb-4d4e-9e9b-4a24c7dbd313/volumes" Nov 01 23:07:14.824830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.824795 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr"] Nov 01 23:07:14.825657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.825640 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8e865675-b66f-4bc2-af3d-98477f386c8e" containerName="payload-processing" Nov 01 23:07:14.825720 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.825659 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e865675-b66f-4bc2-af3d-98477f386c8e" containerName="payload-processing" Nov 01 23:07:14.825720 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.825684 4981 cpu_manager.go:414] "containerMap: removing container" podUID="52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" containerName="payload-pre-processing" Nov 01 23:07:14.825720 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.825690 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" containerName="payload-pre-processing" Nov 01 23:07:14.825828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.825823 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a03d5b-6491-4f44-9ea6-dc6f1e612ca0" containerName="payload-pre-processing" Nov 01 23:07:14.825875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.825842 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e865675-b66f-4bc2-af3d-98477f386c8e" containerName="payload-processing" Nov 01 23:07:14.831187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.831168 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:14.842816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.840211 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:07:14.842816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.840708 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:07:14.847618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:14.843408 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr"] Nov 01 23:07:15.025298 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.025200 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.025298 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.025258 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.025508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.025470 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf7lp\" (UniqueName: \"kubernetes.io/projected/b59b4dad-93de-4dae-a937-7c2c5029d54b-kube-api-access-vf7lp\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.127387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.127349 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf7lp\" (UniqueName: \"kubernetes.io/projected/b59b4dad-93de-4dae-a937-7c2c5029d54b-kube-api-access-vf7lp\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.127639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.127595 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.127763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.127642 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.128523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.128497 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.130144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.130119 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.135392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.135373 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf7lp\" (UniqueName: \"kubernetes.io/projected/b59b4dad-93de-4dae-a937-7c2c5029d54b-kube-api-access-vf7lp\") pod \"maas-api-revoke-sub-7585e6c3-fjntr\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.152809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.152783 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:15.239624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.239322 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:15.318721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.317385 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:15.538723 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:15.538675 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59b4dad_93de_4dae_a937_7c2c5029d54b.slice/crio-a738281c8014180b05db9afeae2efcf1299869ea8a57b108291f0987df869d3b WatchSource:0}: Error finding container a738281c8014180b05db9afeae2efcf1299869ea8a57b108291f0987df869d3b: Status 404 returned error can't find the container with id a738281c8014180b05db9afeae2efcf1299869ea8a57b108291f0987df869d3b Nov 01 23:07:15.541485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.541445 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr"] Nov 01 23:07:15.727095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.727054 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" event={"ID":"b59b4dad-93de-4dae-a937-7c2c5029d54b","Type":"ContainerStarted","Data":"a738281c8014180b05db9afeae2efcf1299869ea8a57b108291f0987df869d3b"} Nov 01 23:07:15.731301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.731273 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:15.732036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.731977 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:15.734951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.734931 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:07:15.947282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:15.947250 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:16.042622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:16.040296 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4"] Nov 01 23:07:16.053628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:16.047198 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f0889c-n6gg4"] Nov 01 23:07:16.737001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:16.736967 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8af5ae3-5412-49ee-b1c6-c790ce6b655d" path="/var/lib/kubelet/pods/b8af5ae3-5412-49ee-b1c6-c790ce6b655d/volumes" Nov 01 23:07:16.737473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:16.737301 4981 generic.go:356] "Generic (PLEG): container finished" podID="b59b4dad-93de-4dae-a937-7c2c5029d54b" containerID="2e5aef08d8d4d5e9a5c35b3dd86f7fd24f6d38e67ff625008e5e184b02b386bc" exitCode=0 Nov 01 23:07:16.737473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:16.737453 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" event={"ID":"b59b4dad-93de-4dae-a937-7c2c5029d54b","Type":"ContainerDied","Data":"2e5aef08d8d4d5e9a5c35b3dd86f7fd24f6d38e67ff625008e5e184b02b386bc"} Nov 01 23:07:16.810147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:16.810119 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:17.553652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.553603 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:17.916134 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.916111 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:17.980969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.980939 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-openshift-ca\") pod \"b59b4dad-93de-4dae-a937-7c2c5029d54b\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " Nov 01 23:07:17.981169 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.981000 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b59b4dad-93de-4dae-a937-7c2c5029d54b-kube-api-access-vf7lp\" (UniqueName: \"kubernetes.io/projected/b59b4dad-93de-4dae-a937-7c2c5029d54b-kube-api-access-vf7lp\") pod \"b59b4dad-93de-4dae-a937-7c2c5029d54b\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " Nov 01 23:07:17.981255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.981166 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-xks-ca\") pod \"b59b4dad-93de-4dae-a937-7c2c5029d54b\" (UID: \"b59b4dad-93de-4dae-a937-7c2c5029d54b\") " Nov 01 23:07:17.981378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.981351 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-openshift-ca" pod "b59b4dad-93de-4dae-a937-7c2c5029d54b" (UID: "b59b4dad-93de-4dae-a937-7c2c5029d54b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:17.981896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.981869 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:17.983380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.983340 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59b4dad-93de-4dae-a937-7c2c5029d54b-kube-api-access-vf7lp" pod "b59b4dad-93de-4dae-a937-7c2c5029d54b" (UID: "b59b4dad-93de-4dae-a937-7c2c5029d54b"). InnerVolumeSpecName "kube-api-access-vf7lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:17.983716 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:17.983685 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-xks-ca" pod "b59b4dad-93de-4dae-a937-7c2c5029d54b" (UID: "b59b4dad-93de-4dae-a937-7c2c5029d54b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:18.083903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:18.083826 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf7lp\" (UniqueName: \"kubernetes.io/projected/b59b4dad-93de-4dae-a937-7c2c5029d54b-kube-api-access-vf7lp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:18.083903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:18.083857 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b59b4dad-93de-4dae-a937-7c2c5029d54b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:18.251625 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:18.251589 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:18.755550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:18.755526 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" Nov 01 23:07:18.755727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:18.755570 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr" event={"ID":"b59b4dad-93de-4dae-a937-7c2c5029d54b","Type":"ContainerDied","Data":"a738281c8014180b05db9afeae2efcf1299869ea8a57b108291f0987df869d3b"} Nov 01 23:07:18.755727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:18.755644 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a738281c8014180b05db9afeae2efcf1299869ea8a57b108291f0987df869d3b" Nov 01 23:07:19.040221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:19.040133 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:19.418204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:19.418121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:19.518635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:19.518601 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:19.743087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:19.743052 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:19.897597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:19.892602 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:20.190989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.190956 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:20.389117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.389084 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t"] Nov 01 23:07:20.390116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.390096 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b59b4dad-93de-4dae-a937-7c2c5029d54b" containerName="revoke-keys" Nov 01 23:07:20.390212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.390118 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b59b4dad-93de-4dae-a937-7c2c5029d54b" containerName="revoke-keys" Nov 01 23:07:20.390269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.390260 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b59b4dad-93de-4dae-a937-7c2c5029d54b" containerName="revoke-keys" Nov 01 23:07:20.394450 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.394432 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.404595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.404149 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t"] Nov 01 23:07:20.449533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.449435 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db45f99-e584-4264-920c-8003869e3ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.449753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.449633 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz8l9\" (UniqueName: \"kubernetes.io/projected/4db45f99-e584-4264-920c-8003869e3ced-kube-api-access-bz8l9\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.449864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.449832 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db45f99-e584-4264-920c-8003869e3ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.490672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.490633 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:07:20.490672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.490666 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:07:20.551501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.551457 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db45f99-e584-4264-920c-8003869e3ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.551728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.551543 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz8l9\" (UniqueName: \"kubernetes.io/projected/4db45f99-e584-4264-920c-8003869e3ced-kube-api-access-bz8l9\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.551728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.551679 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db45f99-e584-4264-920c-8003869e3ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.552251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.552232 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db45f99-e584-4264-920c-8003869e3ced-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.554075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.554055 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db45f99-e584-4264-920c-8003869e3ced-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.559426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.559404 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz8l9\" (UniqueName: \"kubernetes.io/projected/4db45f99-e584-4264-920c-8003869e3ced-kube-api-access-bz8l9\") pod \"maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.727399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.727312 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:20.953875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:20.953834 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:21.076017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:21.075987 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t"] Nov 01 23:07:21.754162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:21.754122 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:21.792663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:21.792623 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" event={"ID":"4db45f99-e584-4264-920c-8003869e3ced","Type":"ContainerStarted","Data":"4e1bed05020a7f950790a3072b1209f9428e5e942ddac7deedd07184faa50fdd"} Nov 01 23:07:21.792812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:21.792685 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" event={"ID":"4db45f99-e584-4264-920c-8003869e3ced","Type":"ContainerStarted","Data":"ae49d419e4c1d70a91157cfcafa64e6606eca5376c460eb54f341481f4242e61"} Nov 01 23:07:22.802672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:22.802622 4981 generic.go:356] "Generic (PLEG): container finished" podID="4db45f99-e584-4264-920c-8003869e3ced" containerID="4e1bed05020a7f950790a3072b1209f9428e5e942ddac7deedd07184faa50fdd" exitCode=0 Nov 01 23:07:22.803067 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:22.802702 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" event={"ID":"4db45f99-e584-4264-920c-8003869e3ced","Type":"ContainerDied","Data":"4e1bed05020a7f950790a3072b1209f9428e5e942ddac7deedd07184faa50fdd"} Nov 01 23:07:22.912973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:22.912945 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:23.073193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:23.073113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:23.745200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:23.745169 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:24.021323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.021295 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:24.119721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.119686 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db45f99-e584-4264-920c-8003869e3ced-kube-api-access-bz8l9\" (UniqueName: \"kubernetes.io/projected/4db45f99-e584-4264-920c-8003869e3ced-kube-api-access-bz8l9\") pod \"4db45f99-e584-4264-920c-8003869e3ced\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " Nov 01 23:07:24.119892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.119816 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db45f99-e584-4264-920c-8003869e3ced-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db45f99-e584-4264-920c-8003869e3ced-maas-api-openshift-ca\") pod \"4db45f99-e584-4264-920c-8003869e3ced\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " Nov 01 23:07:24.119892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.119878 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db45f99-e584-4264-920c-8003869e3ced-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db45f99-e584-4264-920c-8003869e3ced-maas-api-xks-ca\") pod \"4db45f99-e584-4264-920c-8003869e3ced\" (UID: \"4db45f99-e584-4264-920c-8003869e3ced\") " Nov 01 23:07:24.120174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.120151 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db45f99-e584-4264-920c-8003869e3ced-maas-api-openshift-ca" pod "4db45f99-e584-4264-920c-8003869e3ced" (UID: "4db45f99-e584-4264-920c-8003869e3ced"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:24.120672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.120647 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db45f99-e584-4264-920c-8003869e3ced-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:24.121925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.121902 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db45f99-e584-4264-920c-8003869e3ced-maas-api-xks-ca" pod "4db45f99-e584-4264-920c-8003869e3ced" (UID: "4db45f99-e584-4264-920c-8003869e3ced"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:24.122016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.121990 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db45f99-e584-4264-920c-8003869e3ced-kube-api-access-bz8l9" pod "4db45f99-e584-4264-920c-8003869e3ced" (UID: "4db45f99-e584-4264-920c-8003869e3ced"). InnerVolumeSpecName "kube-api-access-bz8l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:24.221841 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.221810 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f"] Nov 01 23:07:24.222635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.222568 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz8l9\" (UniqueName: \"kubernetes.io/projected/4db45f99-e584-4264-920c-8003869e3ced-kube-api-access-bz8l9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:24.222739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.222639 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db45f99-e584-4264-920c-8003869e3ced-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:24.222739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.222691 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4db45f99-e584-4264-920c-8003869e3ced" containerName="revoke-keys" Nov 01 23:07:24.222739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.222707 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db45f99-e584-4264-920c-8003869e3ced" containerName="revoke-keys" Nov 01 23:07:24.222885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.222872 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db45f99-e584-4264-920c-8003869e3ced" containerName="revoke-keys" Nov 01 23:07:24.226986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.226969 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.237899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.237875 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f"] Nov 01 23:07:24.324563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.324477 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zfgr\" (UniqueName: \"kubernetes.io/projected/2cd4b242-9ffe-44a1-a388-4bf781153174-kube-api-access-5zfgr\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.324761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.324697 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.324935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.324916 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.426274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.426242 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zfgr\" (UniqueName: \"kubernetes.io/projected/2cd4b242-9ffe-44a1-a388-4bf781153174-kube-api-access-5zfgr\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.426453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.426378 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.426533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.426519 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.427151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.427128 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.428772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.428738 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.434295 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.434275 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zfgr\" (UniqueName: \"kubernetes.io/projected/2cd4b242-9ffe-44a1-a388-4bf781153174-kube-api-access-5zfgr\") pod \"maas-api-revoke-sub-4167bb24-bqk2f\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.438210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.438189 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:24.546805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.546765 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:24.831816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.831788 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f"] Nov 01 23:07:24.831943 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:24.831821 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd4b242_9ffe_44a1_a388_4bf781153174.slice/crio-a4d2515627381960d03a794415b8eb67c0615cadc5b04b820e3debb99c9bce8d WatchSource:0}: Error finding container a4d2515627381960d03a794415b8eb67c0615cadc5b04b820e3debb99c9bce8d: Status 404 returned error can't find the container with id a4d2515627381960d03a794415b8eb67c0615cadc5b04b820e3debb99c9bce8d Nov 01 23:07:24.848792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.848763 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" event={"ID":"2cd4b242-9ffe-44a1-a388-4bf781153174","Type":"ContainerStarted","Data":"a4d2515627381960d03a794415b8eb67c0615cadc5b04b820e3debb99c9bce8d"} Nov 01 23:07:24.851011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.850969 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" event={"ID":"4db45f99-e584-4264-920c-8003869e3ced","Type":"ContainerDied","Data":"ae49d419e4c1d70a91157cfcafa64e6606eca5376c460eb54f341481f4242e61"} Nov 01 23:07:24.851140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.851023 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae49d419e4c1d70a91157cfcafa64e6606eca5376c460eb54f341481f4242e61" Nov 01 23:07:24.851140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:24.851027 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t" Nov 01 23:07:25.172538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.172510 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd"] Nov 01 23:07:25.181608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.181548 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l"] Nov 01 23:07:25.202764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.202738 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8"] Nov 01 23:07:25.256691 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.256664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:25.368375 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.368335 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:25.591080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.591005 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:25.862187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.862156 4981 generic.go:356] "Generic (PLEG): container finished" podID="2cd4b242-9ffe-44a1-a388-4bf781153174" containerID="33f64f78740dcfe5e8e9da231047c81f6d1ed22bf54c9765d4332da55fcad0ec" exitCode=0 Nov 01 23:07:25.862307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:25.862281 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" event={"ID":"2cd4b242-9ffe-44a1-a388-4bf781153174","Type":"ContainerDied","Data":"33f64f78740dcfe5e8e9da231047c81f6d1ed22bf54c9765d4332da55fcad0ec"} Nov 01 23:07:25.936566 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:25.936532 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd4b242_9ffe_44a1_a388_4bf781153174.slice/crio-conmon-33f64f78740dcfe5e8e9da231047c81f6d1ed22bf54c9765d4332da55fcad0ec.scope\": RecentStats: unable to find data in memory cache]" Nov 01 23:07:26.042650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:26.042617 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls"] Nov 01 23:07:26.047443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:26.047415 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be5ac508-nd6ls"] Nov 01 23:07:26.167632 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:26.167539 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:26.740241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:26.740206 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b680877d-2b1e-47f7-a7fb-65cde573777d" path="/var/lib/kubelet/pods/b680877d-2b1e-47f7-a7fb-65cde573777d/volumes" Nov 01 23:07:26.879813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:26.879781 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:27.092032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.092008 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:27.203328 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.203293 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-xks-ca\") pod \"2cd4b242-9ffe-44a1-a388-4bf781153174\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " Nov 01 23:07:27.203531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.203485 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd4b242-9ffe-44a1-a388-4bf781153174-kube-api-access-5zfgr\" (UniqueName: \"kubernetes.io/projected/2cd4b242-9ffe-44a1-a388-4bf781153174-kube-api-access-5zfgr\") pod \"2cd4b242-9ffe-44a1-a388-4bf781153174\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " Nov 01 23:07:27.203639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.203536 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-openshift-ca\") pod \"2cd4b242-9ffe-44a1-a388-4bf781153174\" (UID: \"2cd4b242-9ffe-44a1-a388-4bf781153174\") " Nov 01 23:07:27.203974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.203944 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-openshift-ca" pod "2cd4b242-9ffe-44a1-a388-4bf781153174" (UID: "2cd4b242-9ffe-44a1-a388-4bf781153174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:27.204359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.204321 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:27.205632 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.205607 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-xks-ca" pod "2cd4b242-9ffe-44a1-a388-4bf781153174" (UID: "2cd4b242-9ffe-44a1-a388-4bf781153174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:27.205832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.205808 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd4b242-9ffe-44a1-a388-4bf781153174-kube-api-access-5zfgr" pod "2cd4b242-9ffe-44a1-a388-4bf781153174" (UID: "2cd4b242-9ffe-44a1-a388-4bf781153174"). InnerVolumeSpecName "kube-api-access-5zfgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:27.305923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.305824 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd4b242-9ffe-44a1-a388-4bf781153174-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:27.305923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.305864 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zfgr\" (UniqueName: \"kubernetes.io/projected/2cd4b242-9ffe-44a1-a388-4bf781153174-kube-api-access-5zfgr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:27.749631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.749565 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:27.881837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.881811 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" Nov 01 23:07:27.882035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.881860 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f" event={"ID":"2cd4b242-9ffe-44a1-a388-4bf781153174","Type":"ContainerDied","Data":"a4d2515627381960d03a794415b8eb67c0615cadc5b04b820e3debb99c9bce8d"} Nov 01 23:07:27.882035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:27.881918 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4d2515627381960d03a794415b8eb67c0615cadc5b04b820e3debb99c9bce8d" Nov 01 23:07:28.474949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:28.474835 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:29.292926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:29.292874 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:29.849702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:29.849670 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:30.272162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:30.272133 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:30.618909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:30.618825 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:30.722480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:30.722448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:31.138453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:31.138419 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:31.815355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:31.815323 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:32.722014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:32.721978 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:33.365393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.365361 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq"] Nov 01 23:07:33.365941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.365640 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" podUID="69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" containerName="maas-api" containerID="cri-o://cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271" gracePeriod=30 Nov 01 23:07:33.404515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.404470 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:33.467671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.467636 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p"] Nov 01 23:07:33.468182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.468153 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" podUID="535c3010-866b-4d89-8100-a08e10a7ea0b" containerName="payload-processing" containerID="cri-o://91ad56dd3fa993ad6e6e130cb4d15acd7312820438ef58734cfbd827cfc4a814" gracePeriod=30 Nov 01 23:07:33.479437 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.479403 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv"] Nov 01 23:07:33.480010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.479722 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" podUID="00e134e2-2bee-4b6d-b70b-82a8866688c5" containerName="payload-pre-processing" containerID="cri-o://02e26adf8941becc587065d42a0812bb4f257bfef94114cf22ebd912c45fcb33" gracePeriod=30 Nov 01 23:07:33.500870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.500840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:33.595999 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.595963 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:33.755620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.739902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:33.870599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.845089 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:33.901725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.899406 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" podUID="00e134e2-2bee-4b6d-b70b-82a8866688c5" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.1.51:9004: connect: connection refused" Nov 01 23:07:33.901725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.899736 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" podUID="535c3010-866b-4d89-8100-a08e10a7ea0b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.1.52:9004: connect: connection refused" Nov 01 23:07:33.984690 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.984618 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4"] Nov 01 23:07:33.996689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:33.993392 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8e3381-fvqp4"] Nov 01 23:07:34.280250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.277078 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:34.287214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.287188 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:07:34.300307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.300255 4981 generic.go:356] "Generic (PLEG): container finished" podID="535c3010-866b-4d89-8100-a08e10a7ea0b" containerID="91ad56dd3fa993ad6e6e130cb4d15acd7312820438ef58734cfbd827cfc4a814" exitCode=0 Nov 01 23:07:34.300808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.300781 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" event={"ID":"535c3010-866b-4d89-8100-a08e10a7ea0b","Type":"ContainerDied","Data":"91ad56dd3fa993ad6e6e130cb4d15acd7312820438ef58734cfbd827cfc4a814"} Nov 01 23:07:34.375168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.375135 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-kube-api-access-wtx7t\" (UniqueName: \"kubernetes.io/projected/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-kube-api-access-wtx7t\") pod \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " Nov 01 23:07:34.376906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.376883 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-metrics-certs\") pod \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " Nov 01 23:07:34.377068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.377052 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-openshift-service-ca\") pod \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " Nov 01 23:07:34.377240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.377227 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-maas-api-tls\") pod \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\" (UID: \"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e\") " Nov 01 23:07:34.378807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.375237 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" event={"ID":"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e","Type":"ContainerDied","Data":"cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271"} Nov 01 23:07:34.378993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.378978 4981 scope.go:122] "RemoveContainer" containerID="cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271" Nov 01 23:07:34.379233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.375178 4981 generic.go:356] "Generic (PLEG): container finished" podID="69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" containerID="cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271" exitCode=0 Nov 01 23:07:34.407626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.379472 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" event={"ID":"69fe4d0e-ec09-4028-a39b-11b2e9b1e72e","Type":"ContainerDied","Data":"30cfe63b9c22c00836e839b415975e63f5481451e0bedd20601e0005dfff9279"} Nov 01 23:07:34.461365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.375457 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq" Nov 01 23:07:34.461365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.453786 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-openshift-service-ca" pod "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" (UID: "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:34.514024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.513993 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:34.610602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.582672 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.706848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.661465 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-kube-api-access-wtx7t" pod "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" (UID: "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e"). InnerVolumeSpecName "kube-api-access-wtx7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:34.706984 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.662634 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-metrics-certs" pod "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" (UID: "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:34.708000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.707744 4981 scope.go:122] "RemoveContainer" containerID="cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271" Nov 01 23:07:34.708204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.680389 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-maas-api-tls" pod "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" (UID: "69fe4d0e-ec09-4028-a39b-11b2e9b1e72e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:34.726903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.726872 4981 generic.go:356] "Generic (PLEG): container finished" podID="00e134e2-2bee-4b6d-b70b-82a8866688c5" containerID="02e26adf8941becc587065d42a0812bb4f257bfef94114cf22ebd912c45fcb33" exitCode=0 Nov 01 23:07:34.731627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.731601 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" event={"ID":"00e134e2-2bee-4b6d-b70b-82a8866688c5","Type":"ContainerDied","Data":"02e26adf8941becc587065d42a0812bb4f257bfef94114cf22ebd912c45fcb33"} Nov 01 23:07:34.731806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.731786 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn"] Nov 01 23:07:34.733160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.733136 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2cd4b242-9ffe-44a1-a388-4bf781153174" containerName="revoke-keys" Nov 01 23:07:34.733289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.733274 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd4b242-9ffe-44a1-a388-4bf781153174" containerName="revoke-keys" Nov 01 23:07:34.733389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.733378 4981 cpu_manager.go:414] "containerMap: removing container" podUID="69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" containerName="maas-api" Nov 01 23:07:34.733492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.733479 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" containerName="maas-api" Nov 01 23:07:34.733927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.733907 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd4b242-9ffe-44a1-a388-4bf781153174" containerName="revoke-keys" Nov 01 23:07:34.734057 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.734045 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" containerName="maas-api" Nov 01 23:07:34.742202 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:34.742154 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271\": container with ID starting with cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271 not found: ID does not exist" containerID="cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271" Nov 01 23:07:34.742315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.742211 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271"} err="failed to get container status \"cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271\": rpc error: code = NotFound desc = could not find container \"cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271\": container with ID starting with cd649facfefb5c1ccde0044f6f290b20e11e1c0af9b621515a17889990cb4271 not found: ID does not exist" Nov 01 23:07:34.763471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.763439 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn"] Nov 01 23:07:34.763665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.763644 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:07:34.764089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.764050 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:07:34.764402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.764384 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.773639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.771416 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:07:34.773639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.772243 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:07:34.782922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.782897 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d9749b-e6a8-433b-8697-67d2a98ae8d0" path="/var/lib/kubelet/pods/90d9749b-e6a8-433b-8697-67d2a98ae8d0/volumes" Nov 01 23:07:34.809533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.809499 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535c3010-866b-4d89-8100-a08e10a7ea0b-kube-api-access-hktvf\" (UniqueName: \"kubernetes.io/projected/535c3010-866b-4d89-8100-a08e10a7ea0b-kube-api-access-hktvf\") pod \"535c3010-866b-4d89-8100-a08e10a7ea0b\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " Nov 01 23:07:34.810360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.810332 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e134e2-2bee-4b6d-b70b-82a8866688c5-kube-api-access-q9n9d\" (UniqueName: \"kubernetes.io/projected/00e134e2-2bee-4b6d-b70b-82a8866688c5-kube-api-access-q9n9d\") pod \"00e134e2-2bee-4b6d-b70b-82a8866688c5\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " Nov 01 23:07:34.815207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.815142 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e134e2-2bee-4b6d-b70b-82a8866688c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e134e2-2bee-4b6d-b70b-82a8866688c5-plugins-config-volume\") pod \"00e134e2-2bee-4b6d-b70b-82a8866688c5\" (UID: \"00e134e2-2bee-4b6d-b70b-82a8866688c5\") " Nov 01 23:07:34.816725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.815647 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e134e2-2bee-4b6d-b70b-82a8866688c5-plugins-config-volume" pod "00e134e2-2bee-4b6d-b70b-82a8866688c5" (UID: "00e134e2-2bee-4b6d-b70b-82a8866688c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:34.817049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.817026 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535c3010-866b-4d89-8100-a08e10a7ea0b-kube-api-access-hktvf" pod "535c3010-866b-4d89-8100-a08e10a7ea0b" (UID: "535c3010-866b-4d89-8100-a08e10a7ea0b"). InnerVolumeSpecName "kube-api-access-hktvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:34.817216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.817180 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq"] Nov 01 23:07:34.817377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.817359 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535c3010-866b-4d89-8100-a08e10a7ea0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c3010-866b-4d89-8100-a08e10a7ea0b-plugins-config-volume\") pod \"535c3010-866b-4d89-8100-a08e10a7ea0b\" (UID: \"535c3010-866b-4d89-8100-a08e10a7ea0b\") " Nov 01 23:07:34.817990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.817966 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w989x\" (UniqueName: \"kubernetes.io/projected/c9027f1c-3f66-4084-b199-bacccc6e0a67-kube-api-access-w989x\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.818252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.818235 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.818637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.818619 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.818967 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.818949 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.819086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.819074 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.819178 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.819164 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00e134e2-2bee-4b6d-b70b-82a8866688c5-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.819262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.819251 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hktvf\" (UniqueName: \"kubernetes.io/projected/535c3010-866b-4d89-8100-a08e10a7ea0b-kube-api-access-hktvf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.819352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.819340 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtx7t\" (UniqueName: \"kubernetes.io/projected/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e-kube-api-access-wtx7t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.821131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.821103 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535c3010-866b-4d89-8100-a08e10a7ea0b-plugins-config-volume" pod "535c3010-866b-4d89-8100-a08e10a7ea0b" (UID: "535c3010-866b-4d89-8100-a08e10a7ea0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:34.821285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.821222 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e134e2-2bee-4b6d-b70b-82a8866688c5-kube-api-access-q9n9d" pod "00e134e2-2bee-4b6d-b70b-82a8866688c5" (UID: "00e134e2-2bee-4b6d-b70b-82a8866688c5"). InnerVolumeSpecName "kube-api-access-q9n9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:34.823681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.823662 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8e3381-68465bc96b-m8xdq"] Nov 01 23:07:34.921548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.921454 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.921741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.921691 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w989x\" (UniqueName: \"kubernetes.io/projected/c9027f1c-3f66-4084-b199-bacccc6e0a67-kube-api-access-w989x\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.921861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.921841 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.922140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.922108 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9n9d\" (UniqueName: \"kubernetes.io/projected/00e134e2-2bee-4b6d-b70b-82a8866688c5-kube-api-access-q9n9d\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.922140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.922136 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535c3010-866b-4d89-8100-a08e10a7ea0b-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:34.925050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.925011 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.925950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.925921 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:34.932549 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:34.932517 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w989x\" (UniqueName: \"kubernetes.io/projected/c9027f1c-3f66-4084-b199-bacccc6e0a67-kube-api-access-w989x\") pod \"maas-api-revoke-sub-98c75f8a-btnvn\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:35.096844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.096752 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:35.461443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.461401 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn"] Nov 01 23:07:35.472808 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:35.472774 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9027f1c_3f66_4084_b199_bacccc6e0a67.slice/crio-11974bb9723b4a3e5a58cd7bcf44cf0c0034bba99e7e2f41ecafa7c490920302 WatchSource:0}: Error finding container 11974bb9723b4a3e5a58cd7bcf44cf0c0034bba99e7e2f41ecafa7c490920302: Status 404 returned error can't find the container with id 11974bb9723b4a3e5a58cd7bcf44cf0c0034bba99e7e2f41ecafa7c490920302 Nov 01 23:07:35.542810 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.542781 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:35.739109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.739070 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" event={"ID":"c9027f1c-3f66-4084-b199-bacccc6e0a67","Type":"ContainerStarted","Data":"11974bb9723b4a3e5a58cd7bcf44cf0c0034bba99e7e2f41ecafa7c490920302"} Nov 01 23:07:35.741808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.741782 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" Nov 01 23:07:35.741808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.741798 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv" event={"ID":"00e134e2-2bee-4b6d-b70b-82a8866688c5","Type":"ContainerDied","Data":"cca11733de83c9aec54c1931aa4bea6c7a6474f0bb3179f0799836a144b2b01e"} Nov 01 23:07:35.742036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.741857 4981 scope.go:122] "RemoveContainer" containerID="02e26adf8941becc587065d42a0812bb4f257bfef94114cf22ebd912c45fcb33" Nov 01 23:07:35.743475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.743453 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" Nov 01 23:07:35.743613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.743469 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p" event={"ID":"535c3010-866b-4d89-8100-a08e10a7ea0b","Type":"ContainerDied","Data":"e881ad170787e20e159fb70a75e0d59dc191e351ff3c8f29dc4ba55857f41666"} Nov 01 23:07:35.757429 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.757405 4981 scope.go:122] "RemoveContainer" containerID="91ad56dd3fa993ad6e6e130cb4d15acd7312820438ef58734cfbd827cfc4a814" Nov 01 23:07:35.772109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.772085 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv"] Nov 01 23:07:35.780920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.780898 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8e3381-77c4bfbdb7-m9gdv"] Nov 01 23:07:35.795550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.795526 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p"] Nov 01 23:07:35.799744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:35.799723 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8e3381-75d4595886-p6l4p"] Nov 01 23:07:36.471036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:36.471006 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:36.738336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:36.738296 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e134e2-2bee-4b6d-b70b-82a8866688c5" path="/var/lib/kubelet/pods/00e134e2-2bee-4b6d-b70b-82a8866688c5/volumes" Nov 01 23:07:36.738910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:36.738887 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="535c3010-866b-4d89-8100-a08e10a7ea0b" path="/var/lib/kubelet/pods/535c3010-866b-4d89-8100-a08e10a7ea0b/volumes" Nov 01 23:07:36.739401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:36.739381 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fe4d0e-ec09-4028-a39b-11b2e9b1e72e" path="/var/lib/kubelet/pods/69fe4d0e-ec09-4028-a39b-11b2e9b1e72e/volumes" Nov 01 23:07:36.755951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:36.755927 4981 generic.go:356] "Generic (PLEG): container finished" podID="c9027f1c-3f66-4084-b199-bacccc6e0a67" containerID="253f575050f5c16a481d22c0ceed138b6d4bde16c4c0f7cab05aa25071c2f579" exitCode=0 Nov 01 23:07:36.756060 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:36.756004 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" event={"ID":"c9027f1c-3f66-4084-b199-bacccc6e0a67","Type":"ContainerDied","Data":"253f575050f5c16a481d22c0ceed138b6d4bde16c4c0f7cab05aa25071c2f579"} Nov 01 23:07:37.411230 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:37.411200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:37.976995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:37.976969 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:38.098029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.097950 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-xks-ca\") pod \"c9027f1c-3f66-4084-b199-bacccc6e0a67\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " Nov 01 23:07:38.098191 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.098077 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9027f1c-3f66-4084-b199-bacccc6e0a67-kube-api-access-w989x\" (UniqueName: \"kubernetes.io/projected/c9027f1c-3f66-4084-b199-bacccc6e0a67-kube-api-access-w989x\") pod \"c9027f1c-3f66-4084-b199-bacccc6e0a67\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " Nov 01 23:07:38.098242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.098219 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-openshift-ca\") pod \"c9027f1c-3f66-4084-b199-bacccc6e0a67\" (UID: \"c9027f1c-3f66-4084-b199-bacccc6e0a67\") " Nov 01 23:07:38.098589 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.098546 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-openshift-ca" pod "c9027f1c-3f66-4084-b199-bacccc6e0a67" (UID: "c9027f1c-3f66-4084-b199-bacccc6e0a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:38.098889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.098870 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:38.100149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.100122 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9027f1c-3f66-4084-b199-bacccc6e0a67-kube-api-access-w989x" pod "c9027f1c-3f66-4084-b199-bacccc6e0a67" (UID: "c9027f1c-3f66-4084-b199-bacccc6e0a67"). InnerVolumeSpecName "kube-api-access-w989x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:38.100149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.100135 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-xks-ca" pod "c9027f1c-3f66-4084-b199-bacccc6e0a67" (UID: "c9027f1c-3f66-4084-b199-bacccc6e0a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:38.199968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.199938 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9027f1c-3f66-4084-b199-bacccc6e0a67-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:38.199968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.199964 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w989x\" (UniqueName: \"kubernetes.io/projected/c9027f1c-3f66-4084-b199-bacccc6e0a67-kube-api-access-w989x\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:38.206916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.206889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:38.774489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.774465 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" Nov 01 23:07:38.774668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.774465 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn" event={"ID":"c9027f1c-3f66-4084-b199-bacccc6e0a67","Type":"ContainerDied","Data":"11974bb9723b4a3e5a58cd7bcf44cf0c0034bba99e7e2f41ecafa7c490920302"} Nov 01 23:07:38.774668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.774591 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11974bb9723b4a3e5a58cd7bcf44cf0c0034bba99e7e2f41ecafa7c490920302" Nov 01 23:07:38.873117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:38.873084 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:39.480890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.480857 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq"] Nov 01 23:07:39.481668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481652 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c9027f1c-3f66-4084-b199-bacccc6e0a67" containerName="revoke-keys" Nov 01 23:07:39.481730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481669 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9027f1c-3f66-4084-b199-bacccc6e0a67" containerName="revoke-keys" Nov 01 23:07:39.481730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481712 4981 cpu_manager.go:414] "containerMap: removing container" podUID="00e134e2-2bee-4b6d-b70b-82a8866688c5" containerName="payload-pre-processing" Nov 01 23:07:39.481730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481718 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e134e2-2bee-4b6d-b70b-82a8866688c5" containerName="payload-pre-processing" Nov 01 23:07:39.481730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481730 4981 cpu_manager.go:414] "containerMap: removing container" podUID="535c3010-866b-4d89-8100-a08e10a7ea0b" containerName="payload-processing" Nov 01 23:07:39.481875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481735 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535c3010-866b-4d89-8100-a08e10a7ea0b" containerName="payload-processing" Nov 01 23:07:39.481909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481881 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e134e2-2bee-4b6d-b70b-82a8866688c5" containerName="payload-pre-processing" Nov 01 23:07:39.481909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481889 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="535c3010-866b-4d89-8100-a08e10a7ea0b" containerName="payload-processing" Nov 01 23:07:39.481909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.481901 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9027f1c-3f66-4084-b199-bacccc6e0a67" containerName="revoke-keys" Nov 01 23:07:39.485844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.485825 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.489948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.489928 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:07:39.490618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.490595 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:07:39.495082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.493824 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs"] Nov 01 23:07:39.506647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.506621 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq"] Nov 01 23:07:39.506785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.506726 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.511538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.511511 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs"] Nov 01 23:07:39.621398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.621354 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.621398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.621400 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhp2\" (UniqueName: \"kubernetes.io/projected/b66e0b29-bc68-41f2-b661-1a5a44ce4596-kube-api-access-sjhp2\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.621696 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.621503 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwvxd\" (UniqueName: \"kubernetes.io/projected/d7347f47-2381-421c-81b5-2b1109615b48-kube-api-access-xwvxd\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.621696 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.621678 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.621830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.621813 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.621892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.621871 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.726475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.726444 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.726698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.726678 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.726766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.726713 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.731370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.731301 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.731505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.731448 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.731505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.731480 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhp2\" (UniqueName: \"kubernetes.io/projected/b66e0b29-bc68-41f2-b661-1a5a44ce4596-kube-api-access-sjhp2\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.731659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.731540 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwvxd\" (UniqueName: \"kubernetes.io/projected/d7347f47-2381-421c-81b5-2b1109615b48-kube-api-access-xwvxd\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.732343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.732306 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.735710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.735677 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.737479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.737431 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.747619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.744946 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhp2\" (UniqueName: \"kubernetes.io/projected/b66e0b29-bc68-41f2-b661-1a5a44ce4596-kube-api-access-sjhp2\") pod \"maas-api-revoke-sub-b3c43142-29bjq\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.747771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.747747 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwvxd\" (UniqueName: \"kubernetes.io/projected/d7347f47-2381-421c-81b5-2b1109615b48-kube-api-access-xwvxd\") pod \"maas-api-revoke-sub-99942e3a-5v4hs\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.748588 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.748549 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:39.808750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.808722 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:39.824184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.824156 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:39.847881 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:39.847854 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:40.157968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.155065 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:40.196859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.196782 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" containerID="cri-o://4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5" gracePeriod=45 Nov 01 23:07:40.204372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.204341 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" containerID="cri-o://1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c" gracePeriod=45 Nov 01 23:07:40.224897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.224869 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" containerID="cri-o://30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca" gracePeriod=45 Nov 01 23:07:40.313304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.313269 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq"] Nov 01 23:07:40.323491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.323470 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs"] Nov 01 23:07:40.325783 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:40.325758 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66e0b29_bc68_41f2_b661_1a5a44ce4596.slice/crio-2422274b5318b8891a642d2fa542748474bc5ab27976f3a33e23d82dfd768d86 WatchSource:0}: Error finding container 2422274b5318b8891a642d2fa542748474bc5ab27976f3a33e23d82dfd768d86: Status 404 returned error can't find the container with id 2422274b5318b8891a642d2fa542748474bc5ab27976f3a33e23d82dfd768d86 Nov 01 23:07:40.326078 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:40.326060 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7347f47_2381_421c_81b5_2b1109615b48.slice/crio-67203baf2afa14900a08b96af5828e8125307c642efaa6ac6178fd14874d3611 WatchSource:0}: Error finding container 67203baf2afa14900a08b96af5828e8125307c642efaa6ac6178fd14874d3611: Status 404 returned error can't find the container with id 67203baf2afa14900a08b96af5828e8125307c642efaa6ac6178fd14874d3611 Nov 01 23:07:40.329830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.329758 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused" Nov 01 23:07:40.598705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.598675 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 23:07:40.601257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.601188 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 23:07:40.610712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.610689 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 23:07:40.682925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.682895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:40.755796 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.755769 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-home\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-home\") pod \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " Nov 01 23:07:40.755922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.755826 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d65681-0a07-4326-be1a-8df4c0fe831d-kube-api-access-sdp66\" (UniqueName: \"kubernetes.io/projected/a9d65681-0a07-4326-be1a-8df4c0fe831d-kube-api-access-sdp66\") pod \"a9d65681-0a07-4326-be1a-8df4c0fe831d\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " Nov 01 23:07:40.755922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.755862 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-model-cache\") pod \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " Nov 01 23:07:40.755922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.755884 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tmp-dir\") pod \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " Nov 01 23:07:40.755922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.755903 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-dshm\") pod \"a9d65681-0a07-4326-be1a-8df4c0fe831d\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " Nov 01 23:07:40.756145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.755934 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-tmp-dir\") pod \"a9d65681-0a07-4326-be1a-8df4c0fe831d\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " Nov 01 23:07:40.756145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.755992 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-model-cache\") pod \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " Nov 01 23:07:40.756145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756040 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-home\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-home\") pod \"a9d65681-0a07-4326-be1a-8df4c0fe831d\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " Nov 01 23:07:40.756145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756071 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tls-certs\") pod \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " Nov 01 23:07:40.756145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756068 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-home" pod "d3b59a77-aff1-4b88-a94b-0bf68219f2a5" (UID: "d3b59a77-aff1-4b88-a94b-0bf68219f2a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.756402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756154 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-dshm\") pod \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " Nov 01 23:07:40.756402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756258 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-home\") pod \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " Nov 01 23:07:40.756402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756321 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-model-cache" pod "bbd6472b-add6-4a3c-8fff-d09e0b182ba1" (UID: "bbd6472b-add6-4a3c-8fff-d09e0b182ba1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.756402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756342 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-home" pod "a9d65681-0a07-4326-be1a-8df4c0fe831d" (UID: "a9d65681-0a07-4326-be1a-8df4c0fe831d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.756402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756354 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-model-cache\") pod \"a9d65681-0a07-4326-be1a-8df4c0fe831d\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " Nov 01 23:07:40.756402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756397 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9d65681-0a07-4326-be1a-8df4c0fe831d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d65681-0a07-4326-be1a-8df4c0fe831d-tls-certs\") pod \"a9d65681-0a07-4326-be1a-8df4c0fe831d\" (UID: \"a9d65681-0a07-4326-be1a-8df4c0fe831d\") " Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756436 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-tmp-dir" pod "a9d65681-0a07-4326-be1a-8df4c0fe831d" (UID: "a9d65681-0a07-4326-be1a-8df4c0fe831d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756440 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-kube-api-access-sjt9k\" (UniqueName: \"kubernetes.io/projected/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-kube-api-access-sjt9k\") pod \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756505 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tls-certs\") pod \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\" (UID: \"d3b59a77-aff1-4b88-a94b-0bf68219f2a5\") " Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756533 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-model-cache" pod "d3b59a77-aff1-4b88-a94b-0bf68219f2a5" (UID: "d3b59a77-aff1-4b88-a94b-0bf68219f2a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756663 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tmp-dir\") pod \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756729 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-kube-api-access-f2cxs\" (UniqueName: \"kubernetes.io/projected/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-kube-api-access-f2cxs\") pod \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756750 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-model-cache" pod "a9d65681-0a07-4326-be1a-8df4c0fe831d" (UID: "a9d65681-0a07-4326-be1a-8df4c0fe831d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756757 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-dshm\") pod \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\" (UID: \"bbd6472b-add6-4a3c-8fff-d09e0b182ba1\") " Nov 01 23:07:40.757101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.756954 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tmp-dir" pod "bbd6472b-add6-4a3c-8fff-d09e0b182ba1" (UID: "bbd6472b-add6-4a3c-8fff-d09e0b182ba1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.757796 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757775 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.757880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757799 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.757880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757812 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.757880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757826 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.757880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757840 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.757880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757861 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.757880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757878 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.758204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.757860 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-home" pod "bbd6472b-add6-4a3c-8fff-d09e0b182ba1" (UID: "bbd6472b-add6-4a3c-8fff-d09e0b182ba1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.758204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.758090 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tmp-dir" pod "d3b59a77-aff1-4b88-a94b-0bf68219f2a5" (UID: "d3b59a77-aff1-4b88-a94b-0bf68219f2a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.759375 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.759349 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-dshm" pod "d3b59a77-aff1-4b88-a94b-0bf68219f2a5" (UID: "d3b59a77-aff1-4b88-a94b-0bf68219f2a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.760567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.760540 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-dshm" pod "a9d65681-0a07-4326-be1a-8df4c0fe831d" (UID: "a9d65681-0a07-4326-be1a-8df4c0fe831d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.760658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.760630 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-dshm" pod "bbd6472b-add6-4a3c-8fff-d09e0b182ba1" (UID: "bbd6472b-add6-4a3c-8fff-d09e0b182ba1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:07:40.760658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.760645 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tls-certs" pod "bbd6472b-add6-4a3c-8fff-d09e0b182ba1" (UID: "bbd6472b-add6-4a3c-8fff-d09e0b182ba1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:40.760751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.760682 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d65681-0a07-4326-be1a-8df4c0fe831d-kube-api-access-sdp66" pod "a9d65681-0a07-4326-be1a-8df4c0fe831d" (UID: "a9d65681-0a07-4326-be1a-8df4c0fe831d"). InnerVolumeSpecName "kube-api-access-sdp66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:40.760751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.760727 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tls-certs" pod "d3b59a77-aff1-4b88-a94b-0bf68219f2a5" (UID: "d3b59a77-aff1-4b88-a94b-0bf68219f2a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:40.760839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.760770 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-kube-api-access-sjt9k" pod "d3b59a77-aff1-4b88-a94b-0bf68219f2a5" (UID: "d3b59a77-aff1-4b88-a94b-0bf68219f2a5"). InnerVolumeSpecName "kube-api-access-sjt9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:40.761844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.761809 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d65681-0a07-4326-be1a-8df4c0fe831d-tls-certs" pod "a9d65681-0a07-4326-be1a-8df4c0fe831d" (UID: "a9d65681-0a07-4326-be1a-8df4c0fe831d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:40.762101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.762079 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-kube-api-access-f2cxs" pod "bbd6472b-add6-4a3c-8fff-d09e0b182ba1" (UID: "bbd6472b-add6-4a3c-8fff-d09e0b182ba1"). InnerVolumeSpecName "kube-api-access-f2cxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:40.834422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.834387 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" event={"ID":"b66e0b29-bc68-41f2-b661-1a5a44ce4596","Type":"ContainerStarted","Data":"2422274b5318b8891a642d2fa542748474bc5ab27976f3a33e23d82dfd768d86"} Nov 01 23:07:40.836327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.836298 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" event={"ID":"d7347f47-2381-421c-81b5-2b1109615b48","Type":"ContainerStarted","Data":"67203baf2afa14900a08b96af5828e8125307c642efaa6ac6178fd14874d3611"} Nov 01 23:07:40.838047 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.838026 4981 generic.go:356] "Generic (PLEG): container finished" podID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerID="4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5" exitCode=0 Nov 01 23:07:40.838154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.838071 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" Nov 01 23:07:40.838154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.838103 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" event={"ID":"a9d65681-0a07-4326-be1a-8df4c0fe831d","Type":"ContainerDied","Data":"4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5"} Nov 01 23:07:40.838264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.838219 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd" event={"ID":"a9d65681-0a07-4326-be1a-8df4c0fe831d","Type":"ContainerDied","Data":"6a817ce4332db83f4d4275596876b853fe59e4d8f56ae850a60913f11565b7fc"} Nov 01 23:07:40.838407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.838276 4981 scope.go:122] "RemoveContainer" containerID="4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5" Nov 01 23:07:40.839670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.839638 4981 generic.go:356] "Generic (PLEG): container finished" podID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerID="30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca" exitCode=0 Nov 01 23:07:40.839774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.839694 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" Nov 01 23:07:40.839833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.839722 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" event={"ID":"bbd6472b-add6-4a3c-8fff-d09e0b182ba1","Type":"ContainerDied","Data":"30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca"} Nov 01 23:07:40.839833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.839809 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8" event={"ID":"bbd6472b-add6-4a3c-8fff-d09e0b182ba1","Type":"ContainerDied","Data":"ca0885f5440b13d5d304c52a2750376bc4dc016ba8525428bae865c5a16b0aff"} Nov 01 23:07:40.841371 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.841349 4981 generic.go:356] "Generic (PLEG): container finished" podID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerID="1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c" exitCode=0 Nov 01 23:07:40.841459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.841392 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" event={"ID":"d3b59a77-aff1-4b88-a94b-0bf68219f2a5","Type":"ContainerDied","Data":"1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c"} Nov 01 23:07:40.841459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.841414 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" event={"ID":"d3b59a77-aff1-4b88-a94b-0bf68219f2a5","Type":"ContainerDied","Data":"39900e65b35eaf3b2579d9bc867fee612a70c965c425033aa3638070e3d7e1d2"} Nov 01 23:07:40.841546 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.841460 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l" Nov 01 23:07:40.851342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.851325 4981 scope.go:122] "RemoveContainer" containerID="4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5" Nov 01 23:07:40.851659 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:40.851637 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5\": container with ID starting with 4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5 not found: ID does not exist" containerID="4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5" Nov 01 23:07:40.851707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.851667 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5"} err="failed to get container status \"4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5\": rpc error: code = NotFound desc = could not find container \"4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5\": container with ID starting with 4d937e86d946552993346ae9da0df4a31eb478bc10213660ce1d3a7ad48ab4b5 not found: ID does not exist" Nov 01 23:07:40.851707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.851683 4981 scope.go:122] "RemoveContainer" containerID="30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859629 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdp66\" (UniqueName: \"kubernetes.io/projected/a9d65681-0a07-4326-be1a-8df4c0fe831d-kube-api-access-sdp66\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859656 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859668 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d65681-0a07-4326-be1a-8df4c0fe831d-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859677 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859685 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859692 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859700 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d65681-0a07-4326-be1a-8df4c0fe831d-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859713 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjt9k\" (UniqueName: \"kubernetes.io/projected/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-kube-api-access-sjt9k\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859727 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b59a77-aff1-4b88-a94b-0bf68219f2a5-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.859739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859741 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2cxs\" (UniqueName: \"kubernetes.io/projected/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-kube-api-access-f2cxs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.860288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.859756 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6472b-add6-4a3c-8fff-d09e0b182ba1-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:40.862177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.862161 4981 scope.go:122] "RemoveContainer" containerID="30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca" Nov 01 23:07:40.862471 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:40.862455 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca\": container with ID starting with 30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca not found: ID does not exist" containerID="30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca" Nov 01 23:07:40.862514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.862478 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca"} err="failed to get container status \"30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca\": rpc error: code = NotFound desc = could not find container \"30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca\": container with ID starting with 30b3928c2fc6a166b4c59cce88ebbe742e887fb4a0dc4254f820a9a76d2560ca not found: ID does not exist" Nov 01 23:07:40.862514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.862493 4981 scope.go:122] "RemoveContainer" containerID="1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c" Nov 01 23:07:40.863396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.863293 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8"] Nov 01 23:07:40.869331 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.869309 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/simulator-w4-2f3705-kserve-7f4c6fdcc6-4gzg8"] Nov 01 23:07:40.873000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.872983 4981 scope.go:122] "RemoveContainer" containerID="1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c" Nov 01 23:07:40.873244 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:40.873228 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c\": container with ID starting with 1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c not found: ID does not exist" containerID="1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c" Nov 01 23:07:40.873303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.873250 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c"} err="failed to get container status \"1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c\": rpc error: code = NotFound desc = could not find container \"1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c\": container with ID starting with 1bba4bcde08df3f404b22e71deb27b4213ad5d28abfc3f86f8b1d9832559d10c not found: ID does not exist" Nov 01 23:07:40.881255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.881235 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd"] Nov 01 23:07:40.885905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.885881 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-2-w4-2f3705-kserve-7854c9d9cf-xhkvd"] Nov 01 23:07:40.893757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.893736 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l"] Nov 01 23:07:40.899417 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:40.899399 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2f3705/distinct-w4-2f3705-kserve-5d5548fb76-p4z5l"] Nov 01 23:07:41.023450 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.023368 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2"] Nov 01 23:07:41.027783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.027758 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1d9592-99qc2"] Nov 01 23:07:41.343357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.343279 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:41.474705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.474587 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:41.578883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.578844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:41.800275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.800237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:41.853642 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.853560 4981 generic.go:356] "Generic (PLEG): container finished" podID="b66e0b29-bc68-41f2-b661-1a5a44ce4596" containerID="6d723fc1f6de6a3e82b1728aae55fc3c0e8aa3770f6bd6782e2689e4d88803e9" exitCode=0 Nov 01 23:07:41.853829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.853647 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" event={"ID":"b66e0b29-bc68-41f2-b661-1a5a44ce4596","Type":"ContainerDied","Data":"6d723fc1f6de6a3e82b1728aae55fc3c0e8aa3770f6bd6782e2689e4d88803e9"} Nov 01 23:07:41.855160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.855133 4981 generic.go:356] "Generic (PLEG): container finished" podID="d7347f47-2381-421c-81b5-2b1109615b48" containerID="c37939650bb717fc70bf96b80d8fa514938d2e223e05e4c4d6beec493091fcda" exitCode=0 Nov 01 23:07:41.855288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:41.855210 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" event={"ID":"d7347f47-2381-421c-81b5-2b1109615b48","Type":"ContainerDied","Data":"c37939650bb717fc70bf96b80d8fa514938d2e223e05e4c4d6beec493091fcda"} Nov 01 23:07:42.201889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.201838 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:42.292639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.292117 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr"] Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293539 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293568 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293639 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293648 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293672 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293680 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293858 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293887 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" containerName="main" Nov 01 23:07:42.294790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.293902 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" containerName="main" Nov 01 23:07:42.328626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.302704 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.328626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.303339 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr"] Nov 01 23:07:42.436007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.435977 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.436154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.436083 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5w7f\" (UniqueName: \"kubernetes.io/projected/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-kube-api-access-g5w7f\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.436215 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.436160 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.537332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.537257 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.537495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.537375 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5w7f\" (UniqueName: \"kubernetes.io/projected/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-kube-api-access-g5w7f\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.537541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.537492 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.538019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.537995 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.539824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.539805 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.545378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.545356 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5w7f\" (UniqueName: \"kubernetes.io/projected/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-kube-api-access-g5w7f\") pod \"maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.636050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.636012 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:42.704945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.704908 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:42.744448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.744409 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe" path="/var/lib/kubelet/pods/2d9b8edf-a3c3-432f-bc7a-9e9ba7fb32fe/volumes" Nov 01 23:07:42.745118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.745073 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d65681-0a07-4326-be1a-8df4c0fe831d" path="/var/lib/kubelet/pods/a9d65681-0a07-4326-be1a-8df4c0fe831d/volumes" Nov 01 23:07:42.745726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.745703 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd6472b-add6-4a3c-8fff-d09e0b182ba1" path="/var/lib/kubelet/pods/bbd6472b-add6-4a3c-8fff-d09e0b182ba1/volumes" Nov 01 23:07:42.746822 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:42.746797 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b59a77-aff1-4b88-a94b-0bf68219f2a5" path="/var/lib/kubelet/pods/d3b59a77-aff1-4b88-a94b-0bf68219f2a5/volumes" Nov 01 23:07:43.072002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.071858 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr"] Nov 01 23:07:43.169555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.169525 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:43.200763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.200119 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:43.289374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.289327 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-xks-ca\") pod \"d7347f47-2381-421c-81b5-2b1109615b48\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " Nov 01 23:07:43.289646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.289625 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-openshift-ca\") pod \"d7347f47-2381-421c-81b5-2b1109615b48\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " Nov 01 23:07:43.289891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.289869 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7347f47-2381-421c-81b5-2b1109615b48-kube-api-access-xwvxd\" (UniqueName: \"kubernetes.io/projected/d7347f47-2381-421c-81b5-2b1109615b48-kube-api-access-xwvxd\") pod \"d7347f47-2381-421c-81b5-2b1109615b48\" (UID: \"d7347f47-2381-421c-81b5-2b1109615b48\") " Nov 01 23:07:43.290932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.290891 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-openshift-ca" pod "d7347f47-2381-421c-81b5-2b1109615b48" (UID: "d7347f47-2381-421c-81b5-2b1109615b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:43.293809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.293744 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7347f47-2381-421c-81b5-2b1109615b48-kube-api-access-xwvxd" pod "d7347f47-2381-421c-81b5-2b1109615b48" (UID: "d7347f47-2381-421c-81b5-2b1109615b48"). InnerVolumeSpecName "kube-api-access-xwvxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:43.294108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.293959 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-xks-ca" pod "d7347f47-2381-421c-81b5-2b1109615b48" (UID: "d7347f47-2381-421c-81b5-2b1109615b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:43.306723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.306698 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:43.391547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.391469 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-openshift-ca\") pod \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " Nov 01 23:07:43.391729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.391683 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-xks-ca\") pod \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " Nov 01 23:07:43.391799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.391778 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66e0b29-bc68-41f2-b661-1a5a44ce4596-kube-api-access-sjhp2\" (UniqueName: \"kubernetes.io/projected/b66e0b29-bc68-41f2-b661-1a5a44ce4596-kube-api-access-sjhp2\") pod \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\" (UID: \"b66e0b29-bc68-41f2-b661-1a5a44ce4596\") " Nov 01 23:07:43.391920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.391898 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-openshift-ca" pod "b66e0b29-bc68-41f2-b661-1a5a44ce4596" (UID: "b66e0b29-bc68-41f2-b661-1a5a44ce4596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:43.392305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.392282 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:43.392305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.392307 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:43.392475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.392320 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwvxd\" (UniqueName: \"kubernetes.io/projected/d7347f47-2381-421c-81b5-2b1109615b48-kube-api-access-xwvxd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:43.392475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.392333 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7347f47-2381-421c-81b5-2b1109615b48-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:43.393827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.393806 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-xks-ca" pod "b66e0b29-bc68-41f2-b661-1a5a44ce4596" (UID: "b66e0b29-bc68-41f2-b661-1a5a44ce4596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:43.393900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.393844 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66e0b29-bc68-41f2-b661-1a5a44ce4596-kube-api-access-sjhp2" pod "b66e0b29-bc68-41f2-b661-1a5a44ce4596" (UID: "b66e0b29-bc68-41f2-b661-1a5a44ce4596"). InnerVolumeSpecName "kube-api-access-sjhp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:43.493240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.493213 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66e0b29-bc68-41f2-b661-1a5a44ce4596-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:43.493240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.493240 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhp2\" (UniqueName: \"kubernetes.io/projected/b66e0b29-bc68-41f2-b661-1a5a44ce4596-kube-api-access-sjhp2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:43.876087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.876047 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:43.877204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.877178 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" event={"ID":"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f","Type":"ContainerStarted","Data":"18b532a730863ed3c625480388f0be4a20c0f649a1d2737fe2adc2311e994279"} Nov 01 23:07:43.877355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.877227 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" event={"ID":"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f","Type":"ContainerStarted","Data":"8db8c35593b5101e39c362921ef0711506305e9f6af143569d6fb16e07940e21"} Nov 01 23:07:43.878838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.878819 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" Nov 01 23:07:43.878974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.878855 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq" event={"ID":"b66e0b29-bc68-41f2-b661-1a5a44ce4596","Type":"ContainerDied","Data":"2422274b5318b8891a642d2fa542748474bc5ab27976f3a33e23d82dfd768d86"} Nov 01 23:07:43.878974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.878898 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2422274b5318b8891a642d2fa542748474bc5ab27976f3a33e23d82dfd768d86" Nov 01 23:07:43.880456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.880440 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" Nov 01 23:07:43.880617 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.880454 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs" event={"ID":"d7347f47-2381-421c-81b5-2b1109615b48","Type":"ContainerDied","Data":"67203baf2afa14900a08b96af5828e8125307c642efaa6ac6178fd14874d3611"} Nov 01 23:07:43.880617 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.880495 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67203baf2afa14900a08b96af5828e8125307c642efaa6ac6178fd14874d3611" Nov 01 23:07:43.896270 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:43.896229 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" podStartSLOduration=1.896216452 podStartE2EDuration="1.896216452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:43.893027601 +0000 UTC m=+2629.320905534" watchObservedRunningTime="2026-11-01 23:07:43.896216452 +0000 UTC m=+2629.324094405" Nov 01 23:07:44.524915 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:07:44.524883 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:07:44.764872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:44.764844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:44.890102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:44.890017 4981 generic.go:356] "Generic (PLEG): container finished" podID="1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" containerID="18b532a730863ed3c625480388f0be4a20c0f649a1d2737fe2adc2311e994279" exitCode=0 Nov 01 23:07:44.890278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:44.890102 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" event={"ID":"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f","Type":"ContainerDied","Data":"18b532a730863ed3c625480388f0be4a20c0f649a1d2737fe2adc2311e994279"} Nov 01 23:07:45.610105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:45.610051 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:46.078373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.078348 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:46.138315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.138281 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-openshift-ca\") pod \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " Nov 01 23:07:46.138514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.138365 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-kube-api-access-g5w7f\" (UniqueName: \"kubernetes.io/projected/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-kube-api-access-g5w7f\") pod \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " Nov 01 23:07:46.138514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.138447 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-xks-ca\") pod \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\" (UID: \"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f\") " Nov 01 23:07:46.138787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.138751 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-openshift-ca" pod "1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" (UID: "1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:07:46.139392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.139365 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:46.140743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.140718 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-kube-api-access-g5w7f" pod "1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" (UID: "1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f"). InnerVolumeSpecName "kube-api-access-g5w7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:07:46.141218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.141192 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-xks-ca" pod "1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" (UID: "1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:07:46.240961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.240927 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:46.240961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.240956 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5w7f\" (UniqueName: \"kubernetes.io/projected/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f-kube-api-access-g5w7f\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:07:46.342852 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.342819 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:46.870067 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.870029 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g"] Nov 01 23:07:46.913934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.913888 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:46.917156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.917132 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" Nov 01 23:07:46.917297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.917166 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr" event={"ID":"1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f","Type":"ContainerDied","Data":"8db8c35593b5101e39c362921ef0711506305e9f6af143569d6fb16e07940e21"} Nov 01 23:07:46.917297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:46.917215 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8db8c35593b5101e39c362921ef0711506305e9f6af143569d6fb16e07940e21" Nov 01 23:07:47.349603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:47.340098 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:47.984759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:47.984727 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:48.873925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:48.873894 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:49.312735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:49.312699 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb"] Nov 01 23:07:49.339867 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:49.339830 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:49.767528 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:49.767499 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:49.876934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:49.876898 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:50.290545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:50.290506 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:50.536603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:50.536465 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:50.776323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:50.776278 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:51.599868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:51.599835 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:52.377140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.377083 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:52.483916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.483881 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4"] Nov 01 23:07:52.484677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484661 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b66e0b29-bc68-41f2-b661-1a5a44ce4596" containerName="revoke-keys" Nov 01 23:07:52.484738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484678 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66e0b29-bc68-41f2-b661-1a5a44ce4596" containerName="revoke-keys" Nov 01 23:07:52.484738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484701 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" containerName="revoke-keys" Nov 01 23:07:52.484738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484707 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" containerName="revoke-keys" Nov 01 23:07:52.484738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484725 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d7347f47-2381-421c-81b5-2b1109615b48" containerName="revoke-keys" Nov 01 23:07:52.484738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484731 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7347f47-2381-421c-81b5-2b1109615b48" containerName="revoke-keys" Nov 01 23:07:52.484903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484866 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7347f47-2381-421c-81b5-2b1109615b48" containerName="revoke-keys" Nov 01 23:07:52.484903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484883 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66e0b29-bc68-41f2-b661-1a5a44ce4596" containerName="revoke-keys" Nov 01 23:07:52.484903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.484892 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" containerName="revoke-keys" Nov 01 23:07:52.489279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.489257 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.507693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.507664 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4"] Nov 01 23:07:52.587647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.587618 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.587813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.587694 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.587813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.587742 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.587813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.587790 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.587954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.587934 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.587993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.587982 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.588101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.588088 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.588151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.588140 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.588260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.588243 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxhb6\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-kube-api-access-pxhb6\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689381 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689430 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689497 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689556 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689660 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxhb6\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-kube-api-access-pxhb6\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689710 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689777 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689787 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-certs\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.689893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689870 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.690378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689906 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.690378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.689928 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.690378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.690056 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-credential-socket\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.690378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.690313 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-data\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.690525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.690415 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istiod-ca-cert\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.692007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.691979 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-envoy\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.692208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.692190 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-podinfo\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.698147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.698123 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-token\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.698859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.698840 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxhb6\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-kube-api-access-pxhb6\") pod \"e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:52.807165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:52.807130 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:53.100718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:53.100634 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4"] Nov 01 23:07:53.109442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:53.109405 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:07:53.109558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:53.109474 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:07:53.109558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:53.109501 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:07:53.170390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:53.170358 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:53.432745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:53.432712 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:53.538420 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:53.538369 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:54.010833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.010797 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" event={"ID":"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32","Type":"ContainerStarted","Data":"46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6"} Nov 01 23:07:54.011274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.010854 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" event={"ID":"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32","Type":"ContainerStarted","Data":"c0d1c72f70e7d3109eebb73e72dcc131d170d15a19d4e6ebf67835c62a3b8a55"} Nov 01 23:07:54.016354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.016321 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:54.034997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.034944 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" podStartSLOduration=2.034929933 podStartE2EDuration="2.034929933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:54.030340744 +0000 UTC m=+2639.458218675" watchObservedRunningTime="2026-11-01 23:07:54.034929933 +0000 UTC m=+2639.462807864" Nov 01 23:07:54.704105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.703904 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:54.808659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.807778 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:54.810228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.810176 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 23:07:54.810465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.810429 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 23:07:54.831588 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:54.831541 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:55.231765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.231733 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:55.727996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.727968 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:55.808054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.808021 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 23:07:55.808230 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.808073 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 23:07:55.825957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.825926 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:55.841238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.841194 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj"] Nov 01 23:07:55.846234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.846212 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:55.853548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.853521 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9d2a4e\"" Nov 01 23:07:55.854141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.853940 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9d2a4e-dockercfg-8lgc7\"" Nov 01 23:07:55.857205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.857184 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj"] Nov 01 23:07:55.911914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.911881 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j"] Nov 01 23:07:55.916926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.916900 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:55.938412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.932690 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j"] Nov 01 23:07:55.977599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.976197 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86676654-36ab-49a6-bbb1-a9a6730d8fa7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:55.977599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.976250 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rjw\" (UniqueName: \"kubernetes.io/projected/3e3ee469-8880-43db-b734-6cecbb10f0e5-kube-api-access-n5rjw\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:55.977599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.976353 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3ee469-8880-43db-b734-6cecbb10f0e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:55.977599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:55.976606 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtjr\" (UniqueName: \"kubernetes.io/projected/86676654-36ab-49a6-bbb1-a9a6730d8fa7-kube-api-access-fjtjr\") pod \"payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:56.149653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.149213 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtjr\" (UniqueName: \"kubernetes.io/projected/86676654-36ab-49a6-bbb1-a9a6730d8fa7-kube-api-access-fjtjr\") pod \"payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:56.149653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.149366 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86676654-36ab-49a6-bbb1-a9a6730d8fa7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:56.149653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.149392 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rjw\" (UniqueName: \"kubernetes.io/projected/3e3ee469-8880-43db-b734-6cecbb10f0e5-kube-api-access-n5rjw\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:56.149653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.149468 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3ee469-8880-43db-b734-6cecbb10f0e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:56.211184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.150528 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3ee469-8880-43db-b734-6cecbb10f0e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:56.211184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.194347 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86676654-36ab-49a6-bbb1-a9a6730d8fa7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:56.249599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.237298 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rjw\" (UniqueName: \"kubernetes.io/projected/3e3ee469-8880-43db-b734-6cecbb10f0e5-kube-api-access-n5rjw\") pod \"payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:56.263648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.251559 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtjr\" (UniqueName: \"kubernetes.io/projected/86676654-36ab-49a6-bbb1-a9a6730d8fa7-kube-api-access-fjtjr\") pod \"payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:56.288602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.275086 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:56.412166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.411805 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:56.479882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.477067 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:56.615073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.614801 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j"] Nov 01 23:07:56.808137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.808101 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 23:07:56.808315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.808154 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 23:07:56.867763 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:07:56.867733 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86676654_36ab_49a6_bbb1_a9a6730d8fa7.slice/crio-ef93237e13ec550fcf9ef729e11f4f5ef9dcdeebc3624c58309230f442a78ee4 WatchSource:0}: Error finding container ef93237e13ec550fcf9ef729e11f4f5ef9dcdeebc3624c58309230f442a78ee4: Status 404 returned error can't find the container with id ef93237e13ec550fcf9ef729e11f4f5ef9dcdeebc3624c58309230f442a78ee4 Nov 01 23:07:56.868613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:56.868561 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj"] Nov 01 23:07:57.163840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.163809 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:57.345069 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.345031 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" event={"ID":"3e3ee469-8880-43db-b734-6cecbb10f0e5","Type":"ContainerStarted","Data":"22741e46b8b2027a51f3ba6436a47913043f62ecba4d0b4057381bdb3d3e2d12"} Nov 01 23:07:57.345536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.345097 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" event={"ID":"3e3ee469-8880-43db-b734-6cecbb10f0e5","Type":"ContainerStarted","Data":"29471182769fa1f05e3713b2cba534ff4166647fc244fb7733bd17991b7ec8c0"} Nov 01 23:07:57.346282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.346261 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:07:57.352854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.351782 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" event={"ID":"86676654-36ab-49a6-bbb1-a9a6730d8fa7","Type":"ContainerStarted","Data":"e3e668d1224fcaea8d4ddc304a1a1baa8532c5d3e7a9b81441dcd1d5f225b336"} Nov 01 23:07:57.352854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.351844 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" event={"ID":"86676654-36ab-49a6-bbb1-a9a6730d8fa7","Type":"ContainerStarted","Data":"ef93237e13ec550fcf9ef729e11f4f5ef9dcdeebc3624c58309230f442a78ee4"} Nov 01 23:07:57.352854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.352028 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:07:57.370623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.369665 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" podStartSLOduration=2.369651767 podStartE2EDuration="2.369651767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:57.366022097 +0000 UTC m=+2642.793900029" watchObservedRunningTime="2026-11-01 23:07:57.369651767 +0000 UTC m=+2642.797529697" Nov 01 23:07:57.410885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.410830 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" podStartSLOduration=2.410816769 podStartE2EDuration="2.410816769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:57.405157003 +0000 UTC m=+2642.833034935" watchObservedRunningTime="2026-11-01 23:07:57.410816769 +0000 UTC m=+2642.838694700" Nov 01 23:07:57.808173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.808137 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 23:07:57.808370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:57.808201 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 23:07:58.162895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:58.162759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:58.808080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:58.808045 4981 patch_prober.go:29] interesting pod/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 01 23:07:58.808484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:58.808096 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 01 23:07:59.141553 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:59.141466 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:59.841742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:59.841707 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:07:59.949957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:59.947029 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:59.949957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:59.948674 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:07:59.949957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:07:59.949842 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:08:00.363774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:00.363743 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:00.532284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:00.532243 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:00.637086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:00.636999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:00.746271 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:00.746239 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t"] Nov 01 23:08:00.749021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:00.748998 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2f3705-df45t"] Nov 01 23:08:00.804996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:00.804967 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:01.141769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:01.141733 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:01.686137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:01.686060 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:01.917455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:01.917415 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" podUID="a5f8dbc0-0979-4803-9322-6e9ac529e96e" containerName="main" containerID="cri-o://9e1294fe7f527ab606818d617e2b62b9dad59a6def5c331bbdbcb823a65ad030" gracePeriod=45 Nov 01 23:08:02.046603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.046535 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" podUID="a5f8dbc0-0979-4803-9322-6e9ac529e96e" containerName="main" probeResult="failure" output="Get \"https://10.134.1.25:8000/ready\": dial tcp 10.134.1.25:8000: connect: connection refused" Nov 01 23:08:02.210663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.209352 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:08:02.298816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.298782 4981 scope.go:122] "RemoveContainer" containerID="9e1294fe7f527ab606818d617e2b62b9dad59a6def5c331bbdbcb823a65ad030" Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.307799 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tmp-dir\") pod \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.307901 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f8dbc0-0979-4803-9322-6e9ac529e96e-kube-api-access-k6j5v\" (UniqueName: \"kubernetes.io/projected/a5f8dbc0-0979-4803-9322-6e9ac529e96e-kube-api-access-k6j5v\") pod \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.308050 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-home\") pod \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.308188 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tls-certs\") pod \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.308237 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-model-cache\") pod \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.308296 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-dshm\") pod \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\" (UID: \"a5f8dbc0-0979-4803-9322-6e9ac529e96e\") " Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.308934 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-home" pod "a5f8dbc0-0979-4803-9322-6e9ac529e96e" (UID: "a5f8dbc0-0979-4803-9322-6e9ac529e96e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.309133 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tmp-dir" pod "a5f8dbc0-0979-4803-9322-6e9ac529e96e" (UID: "a5f8dbc0-0979-4803-9322-6e9ac529e96e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.309229 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:02.309786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.309547 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-model-cache" pod "a5f8dbc0-0979-4803-9322-6e9ac529e96e" (UID: "a5f8dbc0-0979-4803-9322-6e9ac529e96e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:02.313568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.312288 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f8dbc0-0979-4803-9322-6e9ac529e96e-kube-api-access-k6j5v" pod "a5f8dbc0-0979-4803-9322-6e9ac529e96e" (UID: "a5f8dbc0-0979-4803-9322-6e9ac529e96e"). InnerVolumeSpecName "kube-api-access-k6j5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:02.313568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.312356 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-dshm" pod "a5f8dbc0-0979-4803-9322-6e9ac529e96e" (UID: "a5f8dbc0-0979-4803-9322-6e9ac529e96e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:02.313568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.312505 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tls-certs" pod "a5f8dbc0-0979-4803-9322-6e9ac529e96e" (UID: "a5f8dbc0-0979-4803-9322-6e9ac529e96e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:02.398552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.398521 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" event={"ID":"a5f8dbc0-0979-4803-9322-6e9ac529e96e","Type":"ContainerDied","Data":"9e1294fe7f527ab606818d617e2b62b9dad59a6def5c331bbdbcb823a65ad030"} Nov 01 23:08:02.398729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.398568 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" event={"ID":"a5f8dbc0-0979-4803-9322-6e9ac529e96e","Type":"ContainerDied","Data":"201aafc432eccaa73c6afb5946f788f74c7d3cf30c87058d7053d67009ce4169"} Nov 01 23:08:02.398729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.398640 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g" Nov 01 23:08:02.410902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.410879 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:02.410989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.410905 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:02.410989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.410916 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:02.410989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.410925 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6j5v\" (UniqueName: \"kubernetes.io/projected/a5f8dbc0-0979-4803-9322-6e9ac529e96e-kube-api-access-k6j5v\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:02.410989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.410934 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8dbc0-0979-4803-9322-6e9ac529e96e-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:02.435794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.435768 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g"] Nov 01 23:08:02.440559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.440535 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a9f94e/rate-test-model-a9f94e-kserve-5b99bd9c97-88f4g"] Nov 01 23:08:02.632485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.632398 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:02.737835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.737801 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db45f99-e584-4264-920c-8003869e3ced" path="/var/lib/kubelet/pods/4db45f99-e584-4264-920c-8003869e3ced/volumes" Nov 01 23:08:02.738441 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:02.738423 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f8dbc0-0979-4803-9322-6e9ac529e96e" path="/var/lib/kubelet/pods/a5f8dbc0-0979-4803-9322-6e9ac529e96e/volumes" Nov 01 23:08:03.562944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:03.562915 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:03.663223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:03.663194 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:03.767955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:03.767919 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:04.341181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.341135 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" podUID="2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" containerName="main" containerID="cri-o://5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9" gracePeriod=45 Nov 01 23:08:04.384998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.384972 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:04.504985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.504414 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:04.779000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.778971 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:08:04.859053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859016 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-model-cache\") pod \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " Nov 01 23:08:04.859053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859053 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-home\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-home\") pod \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " Nov 01 23:08:04.859324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859094 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tls-certs\") pod \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " Nov 01 23:08:04.859324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859229 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tmp-dir\") pod \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " Nov 01 23:08:04.859324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859271 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-kube-api-access-2tk9s\" (UniqueName: \"kubernetes.io/projected/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-kube-api-access-2tk9s\") pod \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " Nov 01 23:08:04.859324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859323 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-dshm\") pod \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\" (UID: \"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9\") " Nov 01 23:08:04.859567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859338 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-model-cache" pod "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" (UID: "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:04.859567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859350 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-home" pod "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" (UID: "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:04.859975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.859946 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tmp-dir" pod "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" (UID: "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:04.860282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.860256 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:04.860358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.860287 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:04.860358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.860302 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:04.861739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.861712 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-dshm" pod "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" (UID: "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:08:04.861940 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.861916 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tls-certs" pod "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" (UID: "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:04.862063 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.861951 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-kube-api-access-2tk9s" pod "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" (UID: "2a9dd475-cde0-49d0-89aa-6c42bfad2ab9"). InnerVolumeSpecName "kube-api-access-2tk9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:04.962026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.961991 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:04.962026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.962021 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tk9s\" (UniqueName: \"kubernetes.io/projected/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-kube-api-access-2tk9s\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:04.962026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:04.962031 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:05.188238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.188208 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:05.403119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.403088 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:05.424720 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.424688 4981 generic.go:356] "Generic (PLEG): container finished" podID="2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" containerID="5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9" exitCode=0 Nov 01 23:08:05.424876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.424736 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" Nov 01 23:08:05.424876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.424792 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" event={"ID":"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9","Type":"ContainerDied","Data":"5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9"} Nov 01 23:08:05.424876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.424859 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb" event={"ID":"2a9dd475-cde0-49d0-89aa-6c42bfad2ab9","Type":"ContainerDied","Data":"b93ea676dc601758b920acdb4dace66bc68888e33efbed272bbb6813002ddeca"} Nov 01 23:08:05.425005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.424878 4981 scope.go:122] "RemoveContainer" containerID="5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9" Nov 01 23:08:05.453898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.438409 4981 scope.go:122] "RemoveContainer" containerID="5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9" Nov 01 23:08:05.453898 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:08:05.438771 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9\": container with ID starting with 5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9 not found: ID does not exist" containerID="5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9" Nov 01 23:08:05.453898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.438805 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9"} err="failed to get container status \"5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9\": rpc error: code = NotFound desc = could not find container \"5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9\": container with ID starting with 5572e3b289217b943e45039437322457068924d3b89e43217979d40136316de9 not found: ID does not exist" Nov 01 23:08:05.477460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.477406 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb"] Nov 01 23:08:05.483488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:05.483433 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-56b833/rate-test-model-56b833-kserve-64776bd665-5g2kb"] Nov 01 23:08:06.202568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:06.202526 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:06.742364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:06.742333 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" path="/var/lib/kubelet/pods/2a9dd475-cde0-49d0-89aa-6c42bfad2ab9/volumes" Nov 01 23:08:06.988824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:06.988792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:07.932314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:07.932274 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:08.032856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:08.032822 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:08.132900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:08.132863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:08.261401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:08.261315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:08.363102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:08.363076 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:08:09.057342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:09.057314 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:09.371734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:09.371651 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:08:09.959750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:09.959717 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:10.802142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:10.802085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:11.310497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.310465 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:11.335174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.335141 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc"] Nov 01 23:08:11.335659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.335625 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" podUID="9a8ede17-ef50-4622-8ece-b80e35712dfd" containerName="payload-processing" containerID="cri-o://cb034bb4bb4f96d0ef00446f9288f92dad7e04892343ebd3c1613cf6a090fa0c" gracePeriod=30 Nov 01 23:08:11.345352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.345325 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f"] Nov 01 23:08:11.345558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.345535 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" podUID="f2525914-86ca-4583-8478-e185ad487b36" containerName="payload-pre-processing" containerID="cri-o://0c49f1be3186ba0c7da5ba7e99d44b7b069161446398dc730a8e5168465daf43" gracePeriod=30 Nov 01 23:08:11.411022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.410982 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:11.504616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.504547 4981 generic.go:356] "Generic (PLEG): container finished" podID="f2525914-86ca-4583-8478-e185ad487b36" containerID="0c49f1be3186ba0c7da5ba7e99d44b7b069161446398dc730a8e5168465daf43" exitCode=0 Nov 01 23:08:11.504823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.504610 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" event={"ID":"f2525914-86ca-4583-8478-e185ad487b36","Type":"ContainerDied","Data":"0c49f1be3186ba0c7da5ba7e99d44b7b069161446398dc730a8e5168465daf43"} Nov 01 23:08:11.508919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.508893 4981 generic.go:356] "Generic (PLEG): container finished" podID="9a8ede17-ef50-4622-8ece-b80e35712dfd" containerID="cb034bb4bb4f96d0ef00446f9288f92dad7e04892343ebd3c1613cf6a090fa0c" exitCode=0 Nov 01 23:08:11.509292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.509272 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" event={"ID":"9a8ede17-ef50-4622-8ece-b80e35712dfd","Type":"ContainerDied","Data":"cb034bb4bb4f96d0ef00446f9288f92dad7e04892343ebd3c1613cf6a090fa0c"} Nov 01 23:08:11.515297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.515273 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:11.582730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.582646 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr"] Nov 01 23:08:11.587814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.587784 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00185d-bgnvr"] Nov 01 23:08:11.617023 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.616937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:11.664404 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.664377 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:08:11.706696 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.706664 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:08:11.715668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.715623 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:11.841676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.827112 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2525914-86ca-4583-8478-e185ad487b36-kube-api-access-6vfxw\" (UniqueName: \"kubernetes.io/projected/f2525914-86ca-4583-8478-e185ad487b36-kube-api-access-6vfxw\") pod \"f2525914-86ca-4583-8478-e185ad487b36\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " Nov 01 23:08:11.841676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.827385 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8ede17-ef50-4622-8ece-b80e35712dfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8ede17-ef50-4622-8ece-b80e35712dfd-plugins-config-volume\") pod \"9a8ede17-ef50-4622-8ece-b80e35712dfd\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " Nov 01 23:08:11.841676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.827541 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2525914-86ca-4583-8478-e185ad487b36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2525914-86ca-4583-8478-e185ad487b36-plugins-config-volume\") pod \"f2525914-86ca-4583-8478-e185ad487b36\" (UID: \"f2525914-86ca-4583-8478-e185ad487b36\") " Nov 01 23:08:11.841676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.827643 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8ede17-ef50-4622-8ece-b80e35712dfd-kube-api-access-7l7kw\" (UniqueName: \"kubernetes.io/projected/9a8ede17-ef50-4622-8ece-b80e35712dfd-kube-api-access-7l7kw\") pod \"9a8ede17-ef50-4622-8ece-b80e35712dfd\" (UID: \"9a8ede17-ef50-4622-8ece-b80e35712dfd\") " Nov 01 23:08:11.842855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.842819 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8ede17-ef50-4622-8ece-b80e35712dfd-plugins-config-volume" pod "9a8ede17-ef50-4622-8ece-b80e35712dfd" (UID: "9a8ede17-ef50-4622-8ece-b80e35712dfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:11.863844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.863801 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2525914-86ca-4583-8478-e185ad487b36-plugins-config-volume" pod "f2525914-86ca-4583-8478-e185ad487b36" (UID: "f2525914-86ca-4583-8478-e185ad487b36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:11.899806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.899768 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8ede17-ef50-4622-8ece-b80e35712dfd-kube-api-access-7l7kw" pod "9a8ede17-ef50-4622-8ece-b80e35712dfd" (UID: "9a8ede17-ef50-4622-8ece-b80e35712dfd"). InnerVolumeSpecName "kube-api-access-7l7kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:11.900488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.900444 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2525914-86ca-4583-8478-e185ad487b36-kube-api-access-6vfxw" pod "f2525914-86ca-4583-8478-e185ad487b36" (UID: "f2525914-86ca-4583-8478-e185ad487b36"). InnerVolumeSpecName "kube-api-access-6vfxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:11.934659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.933488 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8ede17-ef50-4622-8ece-b80e35712dfd-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:11.934659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.933527 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2525914-86ca-4583-8478-e185ad487b36-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:11.934659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.933543 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7kw\" (UniqueName: \"kubernetes.io/projected/9a8ede17-ef50-4622-8ece-b80e35712dfd-kube-api-access-7l7kw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:11.934659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.933558 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfxw\" (UniqueName: \"kubernetes.io/projected/f2525914-86ca-4583-8478-e185ad487b36-kube-api-access-6vfxw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:11.993586 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:11.993538 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:12.518542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.518512 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" Nov 01 23:08:12.518764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.518509 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f" event={"ID":"f2525914-86ca-4583-8478-e185ad487b36","Type":"ContainerDied","Data":"390d2404d5add9b262af047317f6be5fd10a75f63f6d08285c9d175bf936fd31"} Nov 01 23:08:12.518764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.518675 4981 scope.go:122] "RemoveContainer" containerID="0c49f1be3186ba0c7da5ba7e99d44b7b069161446398dc730a8e5168465daf43" Nov 01 23:08:12.520361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.520332 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" Nov 01 23:08:12.520361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.520337 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc" event={"ID":"9a8ede17-ef50-4622-8ece-b80e35712dfd","Type":"ContainerDied","Data":"ee47fc69e0f44bd901137d60e428f28063846a4468e70e6f8022e44e80485dc2"} Nov 01 23:08:12.532006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.531989 4981 scope.go:122] "RemoveContainer" containerID="cb034bb4bb4f96d0ef00446f9288f92dad7e04892343ebd3c1613cf6a090fa0c" Nov 01 23:08:12.547550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.547524 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f"] Nov 01 23:08:12.552083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.552062 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-00185d-544747887-4st8f"] Nov 01 23:08:12.562995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.562958 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc"] Nov 01 23:08:12.567235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.567211 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-00185d-5f67ffbb8d-25fxc"] Nov 01 23:08:12.683767 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.683734 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:12.769557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.769518 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f" path="/var/lib/kubelet/pods/1a81e9a9-85f7-4f6f-8377-4d8aa2e02d9f/volumes" Nov 01 23:08:12.779459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.779425 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8ede17-ef50-4622-8ece-b80e35712dfd" path="/var/lib/kubelet/pods/9a8ede17-ef50-4622-8ece-b80e35712dfd/volumes" Nov 01 23:08:12.780521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:12.780486 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2525914-86ca-4583-8478-e185ad487b36" path="/var/lib/kubelet/pods/f2525914-86ca-4583-8478-e185ad487b36/volumes" Nov 01 23:08:13.430044 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.430005 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v"] Nov 01 23:08:13.431278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431248 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f2525914-86ca-4583-8478-e185ad487b36" containerName="payload-pre-processing" Nov 01 23:08:13.431278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431277 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2525914-86ca-4583-8478-e185ad487b36" containerName="payload-pre-processing" Nov 01 23:08:13.431476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431296 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" containerName="main" Nov 01 23:08:13.431476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431304 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" containerName="main" Nov 01 23:08:13.431476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431344 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9a8ede17-ef50-4622-8ece-b80e35712dfd" containerName="payload-processing" Nov 01 23:08:13.431476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431351 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8ede17-ef50-4622-8ece-b80e35712dfd" containerName="payload-processing" Nov 01 23:08:13.431476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431399 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a5f8dbc0-0979-4803-9322-6e9ac529e96e" containerName="main" Nov 01 23:08:13.431476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431406 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f8dbc0-0979-4803-9322-6e9ac529e96e" containerName="main" Nov 01 23:08:13.431879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431608 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9dd475-cde0-49d0-89aa-6c42bfad2ab9" containerName="main" Nov 01 23:08:13.431879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431634 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8ede17-ef50-4622-8ece-b80e35712dfd" containerName="payload-processing" Nov 01 23:08:13.431879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431655 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f8dbc0-0979-4803-9322-6e9ac529e96e" containerName="main" Nov 01 23:08:13.431879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.431673 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2525914-86ca-4583-8478-e185ad487b36" containerName="payload-pre-processing" Nov 01 23:08:13.437918 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.437883 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.448185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.448159 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-01e30d-serving-cert\"" Nov 01 23:08:13.460634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.460609 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v"] Nov 01 23:08:13.604408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.562730 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-maas-api-tls\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.604408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.601268 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d12fbc1-585d-4f93-9d26-073c87434b14-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.604408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.601548 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vb2\" (UniqueName: \"kubernetes.io/projected/7d12fbc1-585d-4f93-9d26-073c87434b14-kube-api-access-w4vb2\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.604408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.601136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:13.604408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.601636 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-metrics-certs\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.703198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.703108 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d12fbc1-585d-4f93-9d26-073c87434b14-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.703387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.703272 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vb2\" (UniqueName: \"kubernetes.io/projected/7d12fbc1-585d-4f93-9d26-073c87434b14-kube-api-access-w4vb2\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.703387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.703305 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-metrics-certs\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.703522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.703431 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-maas-api-tls\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.703872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.703846 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d12fbc1-585d-4f93-9d26-073c87434b14-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.705889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.705865 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-metrics-certs\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.705976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.705918 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-maas-api-tls\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.733284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.733252 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vb2\" (UniqueName: \"kubernetes.io/projected/7d12fbc1-585d-4f93-9d26-073c87434b14-kube-api-access-w4vb2\") pod \"maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:13.754308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.754277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:13.755259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:13.755238 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:14.047723 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:14.047693 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d12fbc1_585d_4f93_9d26_073c87434b14.slice/crio-632aa355bd07317593e075c3ff021efc31702bfedfea6b6d5fd0bfd20fec1e22 WatchSource:0}: Error finding container 632aa355bd07317593e075c3ff021efc31702bfedfea6b6d5fd0bfd20fec1e22: Status 404 returned error can't find the container with id 632aa355bd07317593e075c3ff021efc31702bfedfea6b6d5fd0bfd20fec1e22 Nov 01 23:08:14.048657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:14.048631 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v"] Nov 01 23:08:14.457726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:14.457700 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:08:14.482860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:14.482826 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:14.549800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:14.549762 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" event={"ID":"7d12fbc1-585d-4f93-9d26-073c87434b14","Type":"ContainerStarted","Data":"632aa355bd07317593e075c3ff021efc31702bfedfea6b6d5fd0bfd20fec1e22"} Nov 01 23:08:14.559045 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:08:14.559017 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:08:14.632466 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:14.632442 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:08:14.800666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:14.800564 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj_86676654-36ab-49a6-bbb1-a9a6730d8fa7/payload-processing/0.log" Nov 01 23:08:15.169046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.168975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:15.394091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.394055 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm"] Nov 01 23:08:15.395226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.395197 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:15.398978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.398961 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.405331 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.405310 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:08:15.408280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.408258 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:08:15.413455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.413435 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm"] Nov 01 23:08:15.538988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.538956 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.539449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.539150 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.539449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.539186 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llggr\" (UniqueName: \"kubernetes.io/projected/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-kube-api-access-llggr\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.559774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.559741 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" event={"ID":"7d12fbc1-585d-4f93-9d26-073c87434b14","Type":"ContainerStarted","Data":"99baf992d9a64869e780a49b4242e8f0a2c735ff28360f43e68beff8a2b261dd"} Nov 01 23:08:15.559930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.559827 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:15.584427 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.584370 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" podStartSLOduration=2.07860984 podStartE2EDuration="2.584357587s" totalImagesPullingTime="505.747747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:15.580806777 +0000 UTC m=+2661.008684707" watchObservedRunningTime="2026-11-01 23:08:15.584357587 +0000 UTC m=+2661.012235566" Nov 01 23:08:15.640465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.640431 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.640669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.640634 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.640758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.640675 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llggr\" (UniqueName: \"kubernetes.io/projected/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-kube-api-access-llggr\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.641097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.641073 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.643114 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.643086 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.648810 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.648784 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llggr\" (UniqueName: \"kubernetes.io/projected/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-kube-api-access-llggr\") pod \"maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:15.723565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:15.723296 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:16.027809 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:16.027778 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308c4e9d_56a7_4543_9c7b_0f9ff0f15fe3.slice/crio-7b679f2d9a516a8bf1e5ec471d5e9ff56144904ef748cab34d6731da01e0825c WatchSource:0}: Error finding container 7b679f2d9a516a8bf1e5ec471d5e9ff56144904ef748cab34d6731da01e0825c: Status 404 returned error can't find the container with id 7b679f2d9a516a8bf1e5ec471d5e9ff56144904ef748cab34d6731da01e0825c Nov 01 23:08:16.028798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:16.028769 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm"] Nov 01 23:08:16.124553 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:16.124485 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:16.582648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:16.582604 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" event={"ID":"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3","Type":"ContainerStarted","Data":"7b679f2d9a516a8bf1e5ec471d5e9ff56144904ef748cab34d6731da01e0825c"} Nov 01 23:08:17.053741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:17.053711 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:17.594251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:17.594216 4981 generic.go:356] "Generic (PLEG): container finished" podID="308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" containerID="1e8b3df0081394e7f2a1eb258de1a72c6c74f2cf8fb5472854b2b9afa2f2feb1" exitCode=0 Nov 01 23:08:17.594684 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:17.594283 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" event={"ID":"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3","Type":"ContainerDied","Data":"1e8b3df0081394e7f2a1eb258de1a72c6c74f2cf8fb5472854b2b9afa2f2feb1"} Nov 01 23:08:17.796753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:17.796713 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:18.307430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:18.307395 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:18.403662 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:18.403613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:18.504179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:18.504140 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:18.636087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:18.636029 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:18.979476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:18.979448 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:19.002918 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.002888 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-openshift-ca\") pod \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " Nov 01 23:08:19.003104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.002941 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-xks-ca\") pod \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " Nov 01 23:08:19.003104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.003005 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-kube-api-access-llggr\" (UniqueName: \"kubernetes.io/projected/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-kube-api-access-llggr\") pod \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\" (UID: \"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3\") " Nov 01 23:08:19.003311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.003285 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-openshift-ca" pod "308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" (UID: "308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:19.003973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.003949 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:19.005167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.005141 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-kube-api-access-llggr" pod "308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" (UID: "308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3"). InnerVolumeSpecName "kube-api-access-llggr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:19.005228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.005178 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-xks-ca" pod "308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" (UID: "308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:19.105785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.105752 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:19.105785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.105784 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llggr\" (UniqueName: \"kubernetes.io/projected/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3-kube-api-access-llggr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:19.155746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.155713 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp"] Nov 01 23:08:19.156766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.156744 4981 cpu_manager.go:414] "containerMap: removing container" podUID="308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" containerName="revoke-keys" Nov 01 23:08:19.156842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.156770 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" containerName="revoke-keys" Nov 01 23:08:19.156977 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.156967 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" containerName="revoke-keys" Nov 01 23:08:19.161032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.161008 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.211645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.204515 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp"] Nov 01 23:08:19.211645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.206837 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.211645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.206974 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.211645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.207275 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxjk\" (UniqueName: \"kubernetes.io/projected/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-kube-api-access-vcxjk\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.308313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.308260 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxjk\" (UniqueName: \"kubernetes.io/projected/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-kube-api-access-vcxjk\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.308508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.308374 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.308508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.308447 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.309138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.309116 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.311138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.311111 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.318233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.318210 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxjk\" (UniqueName: \"kubernetes.io/projected/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-kube-api-access-vcxjk\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.479843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.479750 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:19.620931 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.620901 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" Nov 01 23:08:19.621118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.620921 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm" event={"ID":"308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3","Type":"ContainerDied","Data":"7b679f2d9a516a8bf1e5ec471d5e9ff56144904ef748cab34d6731da01e0825c"} Nov 01 23:08:19.621118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.620994 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b679f2d9a516a8bf1e5ec471d5e9ff56144904ef748cab34d6731da01e0825c" Nov 01 23:08:19.719766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.719721 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:19.798743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:19.798648 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp"] Nov 01 23:08:19.828920 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:19.828894 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248cc0ad_d1eb_45de_be8d_6013bde1ce0c.slice/crio-45383360909e1f4f2a542114aa5d8753e4fa68b23b50e027e751eb3857ed2f81 WatchSource:0}: Error finding container 45383360909e1f4f2a542114aa5d8753e4fa68b23b50e027e751eb3857ed2f81: Status 404 returned error can't find the container with id 45383360909e1f4f2a542114aa5d8753e4fa68b23b50e027e751eb3857ed2f81 Nov 01 23:08:20.633223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:20.633138 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" event={"ID":"248cc0ad-d1eb-45de-be8d-6013bde1ce0c","Type":"ContainerStarted","Data":"75958c17b008925aa82b111c334031ec4db5da3c3e88412ea051dd75ab63ed62"} Nov 01 23:08:20.633223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:20.633200 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" event={"ID":"248cc0ad-d1eb-45de-be8d-6013bde1ce0c","Type":"ContainerStarted","Data":"45383360909e1f4f2a542114aa5d8753e4fa68b23b50e027e751eb3857ed2f81"} Nov 01 23:08:20.645747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:20.645722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:20.652850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:20.652801 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" podStartSLOduration=1.652784693 podStartE2EDuration="1.652784693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:20.648670585 +0000 UTC m=+2666.076548533" watchObservedRunningTime="2026-11-01 23:08:20.652784693 +0000 UTC m=+2666.080662624" Nov 01 23:08:21.251650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:21.251614 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:21.496534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:21.496505 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:21.589020 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:21.588953 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:08:21.648462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:21.648427 4981 generic.go:356] "Generic (PLEG): container finished" podID="248cc0ad-d1eb-45de-be8d-6013bde1ce0c" containerID="75958c17b008925aa82b111c334031ec4db5da3c3e88412ea051dd75ab63ed62" exitCode=0 Nov 01 23:08:21.648692 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:21.648533 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" event={"ID":"248cc0ad-d1eb-45de-be8d-6013bde1ce0c","Type":"ContainerDied","Data":"75958c17b008925aa82b111c334031ec4db5da3c3e88412ea051dd75ab63ed62"} Nov 01 23:08:21.977239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:21.977202 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:22.658976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.658950 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:22.886149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.886126 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:22.989511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.989481 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-kube-api-access-vcxjk\" (UniqueName: \"kubernetes.io/projected/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-kube-api-access-vcxjk\") pod \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " Nov 01 23:08:22.989745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.989639 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-xks-ca\") pod \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " Nov 01 23:08:22.989824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.989787 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-openshift-ca\") pod \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\" (UID: \"248cc0ad-d1eb-45de-be8d-6013bde1ce0c\") " Nov 01 23:08:22.990220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.990190 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-openshift-ca" pod "248cc0ad-d1eb-45de-be8d-6013bde1ce0c" (UID: "248cc0ad-d1eb-45de-be8d-6013bde1ce0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:22.990800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.990776 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:22.991723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.991705 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-kube-api-access-vcxjk" pod "248cc0ad-d1eb-45de-be8d-6013bde1ce0c" (UID: "248cc0ad-d1eb-45de-be8d-6013bde1ce0c"). InnerVolumeSpecName "kube-api-access-vcxjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:22.991891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:22.991875 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-xks-ca" pod "248cc0ad-d1eb-45de-be8d-6013bde1ce0c" (UID: "248cc0ad-d1eb-45de-be8d-6013bde1ce0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:23.092568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:23.092533 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxjk\" (UniqueName: \"kubernetes.io/projected/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-kube-api-access-vcxjk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:23.092568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:23.092563 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/248cc0ad-d1eb-45de-be8d-6013bde1ce0c-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:23.325685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:23.325531 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:23.666973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:23.666899 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" Nov 01 23:08:23.666973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:23.666940 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp" event={"ID":"248cc0ad-d1eb-45de-be8d-6013bde1ce0c","Type":"ContainerDied","Data":"45383360909e1f4f2a542114aa5d8753e4fa68b23b50e027e751eb3857ed2f81"} Nov 01 23:08:23.667387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:23.666989 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45383360909e1f4f2a542114aa5d8753e4fa68b23b50e027e751eb3857ed2f81" Nov 01 23:08:23.990078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:23.990042 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:24.674291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:24.674251 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:25.461987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:25.461956 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:26.669677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:26.669644 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:27.735965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:27.735929 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:28.547547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:28.547456 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:29.281732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:29.281699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:29.390365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:29.390308 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:30.186201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:30.186172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:31.070827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:31.070797 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:31.771988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:31.771947 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:32.490006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:32.489975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:33.170947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:33.170913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:34.021421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:34.021387 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:34.453462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:34.453438 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:08:34.618998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:34.618970 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:08:34.780637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:34.780538 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8_6c81b530-835e-4ec1-aacb-b05d5b8efc37/payload-processing/0.log" Nov 01 23:08:34.891595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:34.889964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:34.995041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:34.995015 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8_6c81b530-835e-4ec1-aacb-b05d5b8efc37/payload-processing/0.log" Nov 01 23:08:35.293064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.293029 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k"] Nov 01 23:08:35.294348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.294320 4981 cpu_manager.go:414] "containerMap: removing container" podUID="248cc0ad-d1eb-45de-be8d-6013bde1ce0c" containerName="revoke-keys" Nov 01 23:08:35.294348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.294347 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248cc0ad-d1eb-45de-be8d-6013bde1ce0c" containerName="revoke-keys" Nov 01 23:08:35.294592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.294534 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="248cc0ad-d1eb-45de-be8d-6013bde1ce0c" containerName="revoke-keys" Nov 01 23:08:35.301386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.301127 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.332335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.332309 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k"] Nov 01 23:08:35.351706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.351666 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.351870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.351714 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4tl\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-kube-api-access-7c4tl\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.351976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.351944 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.352173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.352147 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.352281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.352191 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560eacaa-6c50-4c64-8650-023dc6c924a6-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.352281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.352223 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.352418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.352286 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.352673 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.352636 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.352791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.352709 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.388232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.388204 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9d2a4e-openshift-default-dockercfg-6kq29\"" Nov 01 23:08:35.453988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.453954 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.453999 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560eacaa-6c50-4c64-8650-023dc6c924a6-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454030 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454100 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454305 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454352 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454398 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454403 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454423 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4tl\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-kube-api-access-7c4tl\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454536 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-certs\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454644 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454729 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-credential-socket\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454753 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-data\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.454858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.454833 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560eacaa-6c50-4c64-8650-023dc6c924a6-istiod-ca-cert\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.456774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.456745 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-envoy\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.456977 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.456960 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-podinfo\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.462675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.462646 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4tl\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-kube-api-access-7c4tl\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.462959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.462943 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-token\") pod \"e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.611024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.610956 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj"] Nov 01 23:08:35.616886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.616866 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.622803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.622770 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:08:35.623088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.623070 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:08:35.628148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.628126 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj"] Nov 01 23:08:35.634362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.634339 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:35.657206 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.657179 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.657338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.657231 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgvl\" (UniqueName: \"kubernetes.io/projected/dc07b2a4-972e-438a-a82c-0a4d617c169f-kube-api-access-8hgvl\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.657415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.657390 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.759039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.759008 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.759299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.759062 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgvl\" (UniqueName: \"kubernetes.io/projected/dc07b2a4-972e-438a-a82c-0a4d617c169f-kube-api-access-8hgvl\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.759417 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.759394 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.759916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.759888 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.761880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.761858 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.768294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.768270 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgvl\" (UniqueName: \"kubernetes.io/projected/dc07b2a4-972e-438a-a82c-0a4d617c169f-kube-api-access-8hgvl\") pod \"maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.854637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.854604 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:35.935482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.934849 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:35.966664 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:35.966631 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560eacaa_6c50_4c64_8650_023dc6c924a6.slice/crio-633ba195a582683c8a803d73c397ef0567f1d3f1bbb78bea3bcc25676a12a143 WatchSource:0}: Error finding container 633ba195a582683c8a803d73c397ef0567f1d3f1bbb78bea3bcc25676a12a143: Status 404 returned error can't find the container with id 633ba195a582683c8a803d73c397ef0567f1d3f1bbb78bea3bcc25676a12a143 Nov 01 23:08:35.966798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.966705 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k"] Nov 01 23:08:35.969511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.969475 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:08:35.969654 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.969556 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:08:35.969654 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:35.969645 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:08:36.238288 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:36.238259 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc07b2a4_972e_438a_a82c_0a4d617c169f.slice/crio-4b2af0cd4d6d588eb49059a2d5468a56648573c6b3f8f7530c0c84417fa1451d WatchSource:0}: Error finding container 4b2af0cd4d6d588eb49059a2d5468a56648573c6b3f8f7530c0c84417fa1451d: Status 404 returned error can't find the container with id 4b2af0cd4d6d588eb49059a2d5468a56648573c6b3f8f7530c0c84417fa1451d Nov 01 23:08:36.242440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.242414 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj"] Nov 01 23:08:36.873271 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.873172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:36.903038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.902352 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" event={"ID":"560eacaa-6c50-4c64-8650-023dc6c924a6","Type":"ContainerStarted","Data":"acb1e0898a35e354a8dd91006fae33c316f7e96c4aa4047c4b8f7c9b02dc1393"} Nov 01 23:08:36.903038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.902428 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" event={"ID":"560eacaa-6c50-4c64-8650-023dc6c924a6","Type":"ContainerStarted","Data":"633ba195a582683c8a803d73c397ef0567f1d3f1bbb78bea3bcc25676a12a143"} Nov 01 23:08:36.908777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.908711 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" event={"ID":"dc07b2a4-972e-438a-a82c-0a4d617c169f","Type":"ContainerStarted","Data":"74e36e22ffdfaf73514ac479a4e549c5fa991ca6ee9794abbe7e43c0eb7aaed3"} Nov 01 23:08:36.908777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.908770 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" event={"ID":"dc07b2a4-972e-438a-a82c-0a4d617c169f","Type":"ContainerStarted","Data":"4b2af0cd4d6d588eb49059a2d5468a56648573c6b3f8f7530c0c84417fa1451d"} Nov 01 23:08:36.969626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.967372 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" podStartSLOduration=1.967349191 podStartE2EDuration="1.967349191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:36.921668038 +0000 UTC m=+2682.349545969" watchObservedRunningTime="2026-11-01 23:08:36.967349191 +0000 UTC m=+2682.395227123" Nov 01 23:08:36.978498 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:36.978443 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" podStartSLOduration=1.978428476 podStartE2EDuration="1.978428476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:36.939061043 +0000 UTC m=+2682.366939010" watchObservedRunningTime="2026-11-01 23:08:36.978428476 +0000 UTC m=+2682.406306453" Nov 01 23:08:37.647611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:37.647484 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:37.648267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:37.648224 4981 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 01 23:08:37.648543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:37.648504 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 01 23:08:37.818822 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:37.818784 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:37.937987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:37.937950 4981 generic.go:356] "Generic (PLEG): container finished" podID="dc07b2a4-972e-438a-a82c-0a4d617c169f" containerID="74e36e22ffdfaf73514ac479a4e549c5fa991ca6ee9794abbe7e43c0eb7aaed3" exitCode=0 Nov 01 23:08:37.938490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:37.938020 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" event={"ID":"dc07b2a4-972e-438a-a82c-0a4d617c169f","Type":"ContainerDied","Data":"74e36e22ffdfaf73514ac479a4e549c5fa991ca6ee9794abbe7e43c0eb7aaed3"} Nov 01 23:08:38.634769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:38.634737 4981 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 01 23:08:38.634954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:38.634786 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 01 23:08:38.667085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:38.667056 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:39.117548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.117499 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:39.209032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.208949 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:39.220914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.220876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:39.312807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.312758 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-openshift-ca\") pod \"dc07b2a4-972e-438a-a82c-0a4d617c169f\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " Nov 01 23:08:39.313022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.312944 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc07b2a4-972e-438a-a82c-0a4d617c169f-kube-api-access-8hgvl\" (UniqueName: \"kubernetes.io/projected/dc07b2a4-972e-438a-a82c-0a4d617c169f-kube-api-access-8hgvl\") pod \"dc07b2a4-972e-438a-a82c-0a4d617c169f\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " Nov 01 23:08:39.313137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.313120 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-xks-ca\") pod \"dc07b2a4-972e-438a-a82c-0a4d617c169f\" (UID: \"dc07b2a4-972e-438a-a82c-0a4d617c169f\") " Nov 01 23:08:39.313201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.313163 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-openshift-ca" pod "dc07b2a4-972e-438a-a82c-0a4d617c169f" (UID: "dc07b2a4-972e-438a-a82c-0a4d617c169f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:39.313790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.313761 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:39.315609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.315566 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-xks-ca" pod "dc07b2a4-972e-438a-a82c-0a4d617c169f" (UID: "dc07b2a4-972e-438a-a82c-0a4d617c169f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:39.315697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.315681 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc07b2a4-972e-438a-a82c-0a4d617c169f-kube-api-access-8hgvl" pod "dc07b2a4-972e-438a-a82c-0a4d617c169f" (UID: "dc07b2a4-972e-438a-a82c-0a4d617c169f"). InnerVolumeSpecName "kube-api-access-8hgvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:39.324713 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.324688 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:39.418618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.417049 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc07b2a4-972e-438a-a82c-0a4d617c169f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:39.418618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.417091 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgvl\" (UniqueName: \"kubernetes.io/projected/dc07b2a4-972e-438a-a82c-0a4d617c169f-kube-api-access-8hgvl\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:39.656705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.655229 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:39.656705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.656066 4981 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 01 23:08:39.656705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.656144 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 01 23:08:39.716649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.716615 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh"] Nov 01 23:08:39.716869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.716841 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" podUID="ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" containerName="payload-processing" containerID="cri-o://b05646d4fa5f925e31e68c90d30e8f995b180e99733e238762f4fdc041b04286" gracePeriod=30 Nov 01 23:08:39.723944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.723915 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6"] Nov 01 23:08:39.724106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.724085 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" podUID="dbec9e08-85fc-454b-b7ac-056410241350" containerName="payload-pre-processing" containerID="cri-o://dc3ce20d40bc812d239b4940d2f1b903255cb54126ea066dd7057e320fa78a16" gracePeriod=30 Nov 01 23:08:39.770158 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.770127 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:39.870534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:39.870502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:40.318840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.255561 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm"] Nov 01 23:08:40.318840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.255667 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c00bbf-5njtm"] Nov 01 23:08:40.318840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.258318 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:40.318840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.316204 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" event={"ID":"dc07b2a4-972e-438a-a82c-0a4d617c169f","Type":"ContainerDied","Data":"4b2af0cd4d6d588eb49059a2d5468a56648573c6b3f8f7530c0c84417fa1451d"} Nov 01 23:08:40.318840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.316251 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2af0cd4d6d588eb49059a2d5468a56648573c6b3f8f7530c0c84417fa1451d" Nov 01 23:08:40.318840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.316307 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj" Nov 01 23:08:40.336916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.333113 4981 generic.go:356] "Generic (PLEG): container finished" podID="dbec9e08-85fc-454b-b7ac-056410241350" containerID="dc3ce20d40bc812d239b4940d2f1b903255cb54126ea066dd7057e320fa78a16" exitCode=0 Nov 01 23:08:40.336916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.333223 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" event={"ID":"dbec9e08-85fc-454b-b7ac-056410241350","Type":"ContainerDied","Data":"dc3ce20d40bc812d239b4940d2f1b903255cb54126ea066dd7057e320fa78a16"} Nov 01 23:08:40.337123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.336965 4981 generic.go:356] "Generic (PLEG): container finished" podID="ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" containerID="b05646d4fa5f925e31e68c90d30e8f995b180e99733e238762f4fdc041b04286" exitCode=0 Nov 01 23:08:40.337123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.337072 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" event={"ID":"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb","Type":"ContainerDied","Data":"b05646d4fa5f925e31e68c90d30e8f995b180e99733e238762f4fdc041b04286"} Nov 01 23:08:40.373476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.371627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:40.465804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.465769 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:08:40.552975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.552949 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:08:40.569519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.569462 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbec9e08-85fc-454b-b7ac-056410241350-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbec9e08-85fc-454b-b7ac-056410241350-plugins-config-volume\") pod \"dbec9e08-85fc-454b-b7ac-056410241350\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " Nov 01 23:08:40.569665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.569636 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-plugins-config-volume\") pod \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " Nov 01 23:08:40.570220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.569906 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-kube-api-access-mnj8w\" (UniqueName: \"kubernetes.io/projected/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-kube-api-access-mnj8w\") pod \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\" (UID: \"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb\") " Nov 01 23:08:40.570220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.569950 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbec9e08-85fc-454b-b7ac-056410241350-kube-api-access-wptjt\" (UniqueName: \"kubernetes.io/projected/dbec9e08-85fc-454b-b7ac-056410241350-kube-api-access-wptjt\") pod \"dbec9e08-85fc-454b-b7ac-056410241350\" (UID: \"dbec9e08-85fc-454b-b7ac-056410241350\") " Nov 01 23:08:40.570220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.570105 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbec9e08-85fc-454b-b7ac-056410241350-plugins-config-volume" pod "dbec9e08-85fc-454b-b7ac-056410241350" (UID: "dbec9e08-85fc-454b-b7ac-056410241350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:40.570220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.570142 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-plugins-config-volume" pod "ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" (UID: "ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:40.570957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.570707 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbec9e08-85fc-454b-b7ac-056410241350-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:40.570957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.570727 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:40.583644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.573706 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-kube-api-access-mnj8w" pod "ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" (UID: "ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb"). InnerVolumeSpecName "kube-api-access-mnj8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:40.607232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.607198 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbec9e08-85fc-454b-b7ac-056410241350-kube-api-access-wptjt" pod "dbec9e08-85fc-454b-b7ac-056410241350" (UID: "dbec9e08-85fc-454b-b7ac-056410241350"). InnerVolumeSpecName "kube-api-access-wptjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:40.607965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.607925 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:40.635447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.635414 4981 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 01 23:08:40.635616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.635468 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 01 23:08:40.672516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.672486 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnj8w\" (UniqueName: \"kubernetes.io/projected/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb-kube-api-access-mnj8w\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:40.672516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.672511 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wptjt\" (UniqueName: \"kubernetes.io/projected/dbec9e08-85fc-454b-b7ac-056410241350-kube-api-access-wptjt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:40.738781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:40.738747 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3" path="/var/lib/kubelet/pods/308c4e9d-56a7-4543-9c7b-0f9ff0f15fe3/volumes" Nov 01 23:08:41.346967 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.346933 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" Nov 01 23:08:41.347434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.346955 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6" event={"ID":"dbec9e08-85fc-454b-b7ac-056410241350","Type":"ContainerDied","Data":"b63988de70931d3c214a22d1a5a396ffce8c0fdb19910d2768720ab16391cbfd"} Nov 01 23:08:41.347434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.347038 4981 scope.go:122] "RemoveContainer" containerID="dc3ce20d40bc812d239b4940d2f1b903255cb54126ea066dd7057e320fa78a16" Nov 01 23:08:41.349380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.349358 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" event={"ID":"ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb","Type":"ContainerDied","Data":"593c9ff2b7a8d88d4db7596b9732524c5713d30be6c28170620d04c93990a1db"} Nov 01 23:08:41.349496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.349431 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh" Nov 01 23:08:41.359630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.359609 4981 scope.go:122] "RemoveContainer" containerID="b05646d4fa5f925e31e68c90d30e8f995b180e99733e238762f4fdc041b04286" Nov 01 23:08:41.372161 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.372141 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6"] Nov 01 23:08:41.403227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.403144 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c00bbf-75dc978d59-p5xg6"] Nov 01 23:08:41.415639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.415612 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh"] Nov 01 23:08:41.420148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.420129 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c00bbf-6648cd49b8-498hh"] Nov 01 23:08:41.635386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.635357 4981 patch_prober.go:29] interesting pod/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 01 23:08:41.635563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.635409 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 01 23:08:41.719473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:41.719439 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:42.585646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:42.585559 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:42.739411 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:42.739374 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" path="/var/lib/kubelet/pods/ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb/volumes" Nov 01 23:08:42.740096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:42.740073 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbec9e08-85fc-454b-b7ac-056410241350" path="/var/lib/kubelet/pods/dbec9e08-85fc-454b-b7ac-056410241350/volumes" Nov 01 23:08:42.966965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:42.966935 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:42.967347 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:42.967319 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:42.969165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:42.969141 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:08:43.471265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:43.471227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:44.278784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:44.278754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:44.525812 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:08:44.525775 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:08:45.091624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:45.091592 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:45.767821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:45.767791 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:46.521283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:46.521256 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:47.389544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:47.389511 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:47.512603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:47.507266 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:47.769789 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:47.769759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:48.208117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.208082 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l"] Nov 01 23:08:48.208918 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.208899 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" containerName="payload-processing" Nov 01 23:08:48.209001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.208921 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" containerName="payload-processing" Nov 01 23:08:48.209001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.208936 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dbec9e08-85fc-454b-b7ac-056410241350" containerName="payload-pre-processing" Nov 01 23:08:48.209001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.208943 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbec9e08-85fc-454b-b7ac-056410241350" containerName="payload-pre-processing" Nov 01 23:08:48.209001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.208951 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dc07b2a4-972e-438a-a82c-0a4d617c169f" containerName="revoke-keys" Nov 01 23:08:48.209001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.208956 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc07b2a4-972e-438a-a82c-0a4d617c169f" containerName="revoke-keys" Nov 01 23:08:48.209257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.209059 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4e9cb9-84dc-43fb-9fc1-02a38671fbeb" containerName="payload-processing" Nov 01 23:08:48.209257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.209067 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbec9e08-85fc-454b-b7ac-056410241350" containerName="payload-pre-processing" Nov 01 23:08:48.209257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.209074 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc07b2a4-972e-438a-a82c-0a4d617c169f" containerName="revoke-keys" Nov 01 23:08:48.215378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.215353 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.220399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.220375 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l"] Nov 01 23:08:48.234910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.234890 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:08:48.235186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.235169 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:08:48.312038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.312010 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:48.390600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.390173 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.390600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.390308 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6hg\" (UniqueName: \"kubernetes.io/projected/a5ab0689-7e04-4a3c-8b91-e07bd287833a-kube-api-access-rb6hg\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.401285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.399979 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.501753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.501674 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.501930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.501765 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.501930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.501853 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6hg\" (UniqueName: \"kubernetes.io/projected/a5ab0689-7e04-4a3c-8b91-e07bd287833a-kube-api-access-rb6hg\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.502445 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.502425 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.504129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.504108 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.511118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.511092 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6hg\" (UniqueName: \"kubernetes.io/projected/a5ab0689-7e04-4a3c-8b91-e07bd287833a-kube-api-access-rb6hg\") pod \"maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.534899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.534872 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:48.819084 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:48.819053 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ab0689_7e04_4a3c_8b91_e07bd287833a.slice/crio-8c99a15b0f65dc1561736f8e8044bc93e1324bef3b8c897b90ff45a47c3186c5 WatchSource:0}: Error finding container 8c99a15b0f65dc1561736f8e8044bc93e1324bef3b8c897b90ff45a47c3186c5: Status 404 returned error can't find the container with id 8c99a15b0f65dc1561736f8e8044bc93e1324bef3b8c897b90ff45a47c3186c5 Nov 01 23:08:48.819346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.819323 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l"] Nov 01 23:08:48.913867 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:48.913833 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:49.059601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.059359 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:49.382357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.382204 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp"] Nov 01 23:08:49.388387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.388356 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-49ab43-rgrvp"] Nov 01 23:08:49.393328 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.393303 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:49.475566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.475487 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:49.491837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.491808 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" event={"ID":"a5ab0689-7e04-4a3c-8b91-e07bd287833a","Type":"ContainerStarted","Data":"0900a78318e6e5946b3ed08c46eaa3aecc2c99e3b3e42f9b3d366a7df8de57f4"} Nov 01 23:08:49.491961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.491854 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" event={"ID":"a5ab0689-7e04-4a3c-8b91-e07bd287833a","Type":"ContainerStarted","Data":"8c99a15b0f65dc1561736f8e8044bc93e1324bef3b8c897b90ff45a47c3186c5"} Nov 01 23:08:49.527599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.522201 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" podStartSLOduration=1.522179301 podStartE2EDuration="1.522179301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:49.512891705 +0000 UTC m=+2694.940769637" watchObservedRunningTime="2026-11-01 23:08:49.522179301 +0000 UTC m=+2694.950057234" Nov 01 23:08:49.839522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:49.839448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:50.177520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.177489 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88"] Nov 01 23:08:50.182134 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.182109 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.187735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.187698 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88"] Nov 01 23:08:50.247524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.247484 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhsf\" (UniqueName: \"kubernetes.io/projected/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-kube-api-access-xkhsf\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.247769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.247745 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.247974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.247957 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.352856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.351172 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.352856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.351305 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhsf\" (UniqueName: \"kubernetes.io/projected/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-kube-api-access-xkhsf\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.352856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.351447 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.352856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.352424 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.355773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.355744 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.367723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.367697 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhsf\" (UniqueName: \"kubernetes.io/projected/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-kube-api-access-xkhsf\") pod \"maas-api-revoke-sub-e09ed389-9pr88\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.502922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.502837 4981 generic.go:356] "Generic (PLEG): container finished" podID="a5ab0689-7e04-4a3c-8b91-e07bd287833a" containerID="0900a78318e6e5946b3ed08c46eaa3aecc2c99e3b3e42f9b3d366a7df8de57f4" exitCode=0 Nov 01 23:08:50.502922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.502878 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" event={"ID":"a5ab0689-7e04-4a3c-8b91-e07bd287833a","Type":"ContainerDied","Data":"0900a78318e6e5946b3ed08c46eaa3aecc2c99e3b3e42f9b3d366a7df8de57f4"} Nov 01 23:08:50.525362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.525331 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:50.594960 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.594489 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:50.738683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.738643 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248cc0ad-d1eb-45de-be8d-6013bde1ce0c" path="/var/lib/kubelet/pods/248cc0ad-d1eb-45de-be8d-6013bde1ce0c/volumes" Nov 01 23:08:50.822566 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:50.822531 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea8e03b_ddcb_4fc0_b09d_c9fb816315f0.slice/crio-ba1ee2e96248d2a72d84cd10c7082ebfc443babb01388152930879254bf32c18 WatchSource:0}: Error finding container ba1ee2e96248d2a72d84cd10c7082ebfc443babb01388152930879254bf32c18: Status 404 returned error can't find the container with id ba1ee2e96248d2a72d84cd10c7082ebfc443babb01388152930879254bf32c18 Nov 01 23:08:50.825134 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:50.825109 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88"] Nov 01 23:08:51.403162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.403081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:51.513927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.513894 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" event={"ID":"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0","Type":"ContainerStarted","Data":"e378f8f523f97f91014d4627fbcbba1fc4b8d27bfee3aee0991bc0719520b94b"} Nov 01 23:08:51.514340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.513951 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" event={"ID":"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0","Type":"ContainerStarted","Data":"ba1ee2e96248d2a72d84cd10c7082ebfc443babb01388152930879254bf32c18"} Nov 01 23:08:51.536736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.536689 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" podStartSLOduration=1.536675528 podStartE2EDuration="1.536675528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:51.5307254 +0000 UTC m=+2696.958603334" watchObservedRunningTime="2026-11-01 23:08:51.536675528 +0000 UTC m=+2696.964553493" Nov 01 23:08:51.681245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.681221 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:51.778875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.778846 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-xks-ca\") pod \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " Nov 01 23:08:51.779143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.779125 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-openshift-ca\") pod \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " Nov 01 23:08:51.779219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.779175 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ab0689-7e04-4a3c-8b91-e07bd287833a-kube-api-access-rb6hg\" (UniqueName: \"kubernetes.io/projected/a5ab0689-7e04-4a3c-8b91-e07bd287833a-kube-api-access-rb6hg\") pod \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\" (UID: \"a5ab0689-7e04-4a3c-8b91-e07bd287833a\") " Nov 01 23:08:51.779510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.779488 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-openshift-ca" pod "a5ab0689-7e04-4a3c-8b91-e07bd287833a" (UID: "a5ab0689-7e04-4a3c-8b91-e07bd287833a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:51.779921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.779897 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:51.780910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.780885 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-xks-ca" pod "a5ab0689-7e04-4a3c-8b91-e07bd287833a" (UID: "a5ab0689-7e04-4a3c-8b91-e07bd287833a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:51.781329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.781313 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ab0689-7e04-4a3c-8b91-e07bd287833a-kube-api-access-rb6hg" pod "a5ab0689-7e04-4a3c-8b91-e07bd287833a" (UID: "a5ab0689-7e04-4a3c-8b91-e07bd287833a"). InnerVolumeSpecName "kube-api-access-rb6hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:51.881721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.881692 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6hg\" (UniqueName: \"kubernetes.io/projected/a5ab0689-7e04-4a3c-8b91-e07bd287833a-kube-api-access-rb6hg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:51.881721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:51.881719 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5ab0689-7e04-4a3c-8b91-e07bd287833a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:52.079949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:52.079906 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:52.523719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:52.523686 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" Nov 01 23:08:52.524152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:52.523720 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l" event={"ID":"a5ab0689-7e04-4a3c-8b91-e07bd287833a","Type":"ContainerDied","Data":"8c99a15b0f65dc1561736f8e8044bc93e1324bef3b8c897b90ff45a47c3186c5"} Nov 01 23:08:52.524152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:52.523767 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c99a15b0f65dc1561736f8e8044bc93e1324bef3b8c897b90ff45a47c3186c5" Nov 01 23:08:52.525433 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:52.525403 4981 generic.go:356] "Generic (PLEG): container finished" podID="bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" containerID="e378f8f523f97f91014d4627fbcbba1fc4b8d27bfee3aee0991bc0719520b94b" exitCode=0 Nov 01 23:08:52.525620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:52.525481 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" event={"ID":"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0","Type":"ContainerDied","Data":"e378f8f523f97f91014d4627fbcbba1fc4b8d27bfee3aee0991bc0719520b94b"} Nov 01 23:08:52.989629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:52.989451 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:53.040477 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.040444 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh"] Nov 01 23:08:53.051863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.051835 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-babd16ca-zrjrh"] Nov 01 23:08:53.713876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.713841 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:53.741447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.741404 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:53.755050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.755024 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c"] Nov 01 23:08:53.755823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.755807 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a5ab0689-7e04-4a3c-8b91-e07bd287833a" containerName="revoke-keys" Nov 01 23:08:53.755869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.755824 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ab0689-7e04-4a3c-8b91-e07bd287833a" containerName="revoke-keys" Nov 01 23:08:53.755869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.755864 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" containerName="revoke-keys" Nov 01 23:08:53.755951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.755870 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" containerName="revoke-keys" Nov 01 23:08:53.755997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.755987 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ab0689-7e04-4a3c-8b91-e07bd287833a" containerName="revoke-keys" Nov 01 23:08:53.756040 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.756006 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" containerName="revoke-keys" Nov 01 23:08:53.760003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.759981 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:53.789179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.789098 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-01e30d\"" Nov 01 23:08:53.795297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.795269 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km"] Nov 01 23:08:53.807968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.807942 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c"] Nov 01 23:08:53.808146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.808005 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:53.809562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.809535 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-openshift-ca\") pod \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " Nov 01 23:08:53.810342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.810315 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-xks-ca\") pod \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " Nov 01 23:08:53.811010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.810730 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-kube-api-access-xkhsf\" (UniqueName: \"kubernetes.io/projected/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-kube-api-access-xkhsf\") pod \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\" (UID: \"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0\") " Nov 01 23:08:53.811099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.811012 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-openshift-ca" pod "bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" (UID: "bea8e03b-ddcb-4fc0-b09d-c9fb816315f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:08:53.815351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.811435 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx488\" (UniqueName: \"kubernetes.io/projected/a90bbf83-c043-4699-a8b1-22c63a1b1894-kube-api-access-mx488\") pod \"payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:53.815351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.811793 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90bbf83-c043-4699-a8b1-22c63a1b1894-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:53.815351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.812155 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.815351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.814056 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-kube-api-access-xkhsf" pod "bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" (UID: "bea8e03b-ddcb-4fc0-b09d-c9fb816315f0"). InnerVolumeSpecName "kube-api-access-xkhsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:08:53.822060 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.821998 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km"] Nov 01 23:08:53.823134 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.823106 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-xks-ca" pod "bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" (UID: "bea8e03b-ddcb-4fc0-b09d-c9fb816315f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:08:53.839369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.839343 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:53.941601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.940759 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90bbf83-c043-4699-a8b1-22c63a1b1894-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:53.941601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.941002 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx488\" (UniqueName: \"kubernetes.io/projected/a90bbf83-c043-4699-a8b1-22c63a1b1894-kube-api-access-mx488\") pod \"payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:53.941601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.941122 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhsf\" (UniqueName: \"kubernetes.io/projected/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-kube-api-access-xkhsf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.941601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.941141 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:08:53.965599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.942444 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90bbf83-c043-4699-a8b1-22c63a1b1894-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:53.975266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:53.975234 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx488\" (UniqueName: \"kubernetes.io/projected/a90bbf83-c043-4699-a8b1-22c63a1b1894-kube-api-access-mx488\") pod \"payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:54.051656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.050857 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b767b8a3-2e0c-464c-b56f-b165274f06c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:54.051656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.050992 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrspp\" (UniqueName: \"kubernetes.io/projected/b767b8a3-2e0c-464c-b56f-b165274f06c3-kube-api-access-nrspp\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:54.087603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.086892 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:54.153014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.152967 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b767b8a3-2e0c-464c-b56f-b165274f06c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:54.153190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.153111 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrspp\" (UniqueName: \"kubernetes.io/projected/b767b8a3-2e0c-464c-b56f-b165274f06c3-kube-api-access-nrspp\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:54.153805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.153785 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b767b8a3-2e0c-464c-b56f-b165274f06c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:54.163850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.163512 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrspp\" (UniqueName: \"kubernetes.io/projected/b767b8a3-2e0c-464c-b56f-b165274f06c3-kube-api-access-nrspp\") pod \"payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:54.169928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.168774 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:54.288763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.288692 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:54.407132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.402019 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c"] Nov 01 23:08:54.407132 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:54.402416 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90bbf83_c043_4699_a8b1_22c63a1b1894.slice/crio-b6fc4c24f73843cc40ca48ba5ed3638a6128600d84192a73eca19af39b0c6d8e WatchSource:0}: Error finding container b6fc4c24f73843cc40ca48ba5ed3638a6128600d84192a73eca19af39b0c6d8e: Status 404 returned error can't find the container with id b6fc4c24f73843cc40ca48ba5ed3638a6128600d84192a73eca19af39b0c6d8e Nov 01 23:08:54.480440 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:54.480414 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb767b8a3_2e0c_464c_b56f_b165274f06c3.slice/crio-2515a8e0b4386588eb1db3819299e4b84d66f2f8122e567e45895a41f04c76f5 WatchSource:0}: Error finding container 2515a8e0b4386588eb1db3819299e4b84d66f2f8122e567e45895a41f04c76f5: Status 404 returned error can't find the container with id 2515a8e0b4386588eb1db3819299e4b84d66f2f8122e567e45895a41f04c76f5 Nov 01 23:08:54.481129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.481109 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km"] Nov 01 23:08:54.546059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.546028 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" event={"ID":"b767b8a3-2e0c-464c-b56f-b165274f06c3","Type":"ContainerStarted","Data":"2515a8e0b4386588eb1db3819299e4b84d66f2f8122e567e45895a41f04c76f5"} Nov 01 23:08:54.547542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.547521 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" Nov 01 23:08:54.547682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.547522 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88" event={"ID":"bea8e03b-ddcb-4fc0-b09d-c9fb816315f0","Type":"ContainerDied","Data":"ba1ee2e96248d2a72d84cd10c7082ebfc443babb01388152930879254bf32c18"} Nov 01 23:08:54.547682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.547628 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1ee2e96248d2a72d84cd10c7082ebfc443babb01388152930879254bf32c18" Nov 01 23:08:54.550401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.549547 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" event={"ID":"a90bbf83-c043-4699-a8b1-22c63a1b1894","Type":"ContainerStarted","Data":"b3a8d9aaf95b1a7d000671d74739a0a68ff9faf09a67cdfc17d9e9ef31e57bfa"} Nov 01 23:08:54.550401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.549636 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" event={"ID":"a90bbf83-c043-4699-a8b1-22c63a1b1894","Type":"ContainerStarted","Data":"b6fc4c24f73843cc40ca48ba5ed3638a6128600d84192a73eca19af39b0c6d8e"} Nov 01 23:08:54.550401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.549721 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:08:54.577014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.576962 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" podStartSLOduration=1.576945043 podStartE2EDuration="1.576945043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:54.572365634 +0000 UTC m=+2700.000243568" watchObservedRunningTime="2026-11-01 23:08:54.576945043 +0000 UTC m=+2700.004822975" Nov 01 23:08:54.783455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.783414 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfacbf6-c481-4776-b4d5-ba300c2ad1cc" path="/var/lib/kubelet/pods/fdfacbf6-c481-4776-b4d5-ba300c2ad1cc/volumes" Nov 01 23:08:54.989472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:54.989440 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:55.561208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:55.561169 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" event={"ID":"b767b8a3-2e0c-464c-b56f-b165274f06c3","Type":"ContainerStarted","Data":"8ff289f6a766f68364d5bd55fef255a8d8ddb4ef8fa82a0b1e8cf167c47f2c45"} Nov 01 23:08:55.561395 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:55.561239 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:08:55.581293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:55.581236 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" podStartSLOduration=2.581219994 podStartE2EDuration="2.581219994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:55.579364231 +0000 UTC m=+2701.007242182" watchObservedRunningTime="2026-11-01 23:08:55.581219994 +0000 UTC m=+2701.009097926" Nov 01 23:08:55.772708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:55.772676 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:56.455431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:56.455392 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:56.561669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:56.560289 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:56.668610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:56.663091 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:56.859932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:56.859847 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4"] Nov 01 23:08:56.860106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:56.860084 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" containerID="cri-o://69ba2e04743cdf97c483a88633c92dc9424b07e253415785713e3ba7431486c1" gracePeriod=30 Nov 01 23:08:56.892053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:56.892019 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:57.377708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.377673 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6"] Nov 01 23:08:57.383513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.383494 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.392798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.392769 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6"] Nov 01 23:08:57.420034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.420007 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:08:57.420182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.420169 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:08:57.435209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.435182 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.435332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.435223 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjdf\" (UniqueName: \"kubernetes.io/projected/b750fd83-0d02-40f4-9adf-fdfa3d885634-kube-api-access-trjdf\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.435332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.435306 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.439224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.439197 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:57.538217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.538184 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.538765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.538496 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.538765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.538545 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjdf\" (UniqueName: \"kubernetes.io/projected/b750fd83-0d02-40f4-9adf-fdfa3d885634-kube-api-access-trjdf\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.539030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.539004 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.541175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.541153 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.546378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.546356 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjdf\" (UniqueName: \"kubernetes.io/projected/b750fd83-0d02-40f4-9adf-fdfa3d885634-kube-api-access-trjdf\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.708807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.708777 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:08:57.995554 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:08:57.995520 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb750fd83_0d02_40f4_9adf_fdfa3d885634.slice/crio-87ef1864cee9828d8b9eea15189838979901bf925ca0fd165cf2d8cb5e77acdc WatchSource:0}: Error finding container 87ef1864cee9828d8b9eea15189838979901bf925ca0fd165cf2d8cb5e77acdc: Status 404 returned error can't find the container with id 87ef1864cee9828d8b9eea15189838979901bf925ca0fd165cf2d8cb5e77acdc Nov 01 23:08:57.997555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:57.997526 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6"] Nov 01 23:08:58.388284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:58.388250 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:58.599476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:58.599389 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" event={"ID":"b750fd83-0d02-40f4-9adf-fdfa3d885634","Type":"ContainerStarted","Data":"21b5826ee32d4c5472fb31414d10a34ea07c1f9ff0215aeaa95432049e108d78"} Nov 01 23:08:58.599476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:58.599448 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" event={"ID":"b750fd83-0d02-40f4-9adf-fdfa3d885634","Type":"ContainerStarted","Data":"87ef1864cee9828d8b9eea15189838979901bf925ca0fd165cf2d8cb5e77acdc"} Nov 01 23:08:58.619066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:58.619020 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" podStartSLOduration=1.619007689 podStartE2EDuration="1.619007689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:08:58.616217971 +0000 UTC m=+2704.044095902" watchObservedRunningTime="2026-11-01 23:08:58.619007689 +0000 UTC m=+2704.046885619" Nov 01 23:08:59.405094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:59.405060 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:08:59.616772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:59.616733 4981 generic.go:356] "Generic (PLEG): container finished" podID="b750fd83-0d02-40f4-9adf-fdfa3d885634" containerID="21b5826ee32d4c5472fb31414d10a34ea07c1f9ff0215aeaa95432049e108d78" exitCode=0 Nov 01 23:08:59.617266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:08:59.616855 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" event={"ID":"b750fd83-0d02-40f4-9adf-fdfa3d885634","Type":"ContainerDied","Data":"21b5826ee32d4c5472fb31414d10a34ea07c1f9ff0215aeaa95432049e108d78"} Nov 01 23:09:00.029043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.029006 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:00.660182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.660153 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:00.849223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.849198 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:09:00.902083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.901996 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b750fd83-0d02-40f4-9adf-fdfa3d885634-kube-api-access-trjdf\" (UniqueName: \"kubernetes.io/projected/b750fd83-0d02-40f4-9adf-fdfa3d885634-kube-api-access-trjdf\") pod \"b750fd83-0d02-40f4-9adf-fdfa3d885634\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " Nov 01 23:09:00.902233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.902193 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-openshift-ca\") pod \"b750fd83-0d02-40f4-9adf-fdfa3d885634\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " Nov 01 23:09:00.902311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.902292 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-xks-ca\") pod \"b750fd83-0d02-40f4-9adf-fdfa3d885634\" (UID: \"b750fd83-0d02-40f4-9adf-fdfa3d885634\") " Nov 01 23:09:00.902610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.902553 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-openshift-ca" pod "b750fd83-0d02-40f4-9adf-fdfa3d885634" (UID: "b750fd83-0d02-40f4-9adf-fdfa3d885634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:00.903075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.903051 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:00.904207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.904183 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b750fd83-0d02-40f4-9adf-fdfa3d885634-kube-api-access-trjdf" pod "b750fd83-0d02-40f4-9adf-fdfa3d885634" (UID: "b750fd83-0d02-40f4-9adf-fdfa3d885634"). InnerVolumeSpecName "kube-api-access-trjdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:00.904389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:00.904375 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-xks-ca" pod "b750fd83-0d02-40f4-9adf-fdfa3d885634" (UID: "b750fd83-0d02-40f4-9adf-fdfa3d885634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:01.004868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:01.004830 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trjdf\" (UniqueName: \"kubernetes.io/projected/b750fd83-0d02-40f4-9adf-fdfa3d885634-kube-api-access-trjdf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:01.004868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:01.004859 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b750fd83-0d02-40f4-9adf-fdfa3d885634-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:01.375946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:01.375910 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:01.637400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:01.637306 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" Nov 01 23:09:01.637400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:01.637328 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6" event={"ID":"b750fd83-0d02-40f4-9adf-fdfa3d885634","Type":"ContainerDied","Data":"87ef1864cee9828d8b9eea15189838979901bf925ca0fd165cf2d8cb5e77acdc"} Nov 01 23:09:01.637400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:01.637381 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ef1864cee9828d8b9eea15189838979901bf925ca0fd165cf2d8cb5e77acdc" Nov 01 23:09:02.062945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.062913 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:02.204971 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.204945 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 23:09:02.365086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365051 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istiod-ca-cert\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365099 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-podinfo\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365146 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-credential-socket\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365214 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-certs\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365243 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-kube-api-access-k9rsb\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-kube-api-access-k9rsb\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365275 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-envoy\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365382 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-socket\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365500 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-data\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365552 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-credential-socket" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:02.365831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365635 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-token\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-token\") pod \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\" (UID: \"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b\") " Nov 01 23:09:02.365831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365644 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istiod-ca-cert" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:02.366029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365853 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-data" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:02.366029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365891 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-certs" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:02.366029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.365803 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-socket" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:02.366518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.366497 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.366562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.366529 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.366562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.366546 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.366782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.366562 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.366782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.366591 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.367814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.367794 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-token" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:02.367920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.367838 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-kube-api-access-k9rsb" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "kube-api-access-k9rsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:02.368030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.368010 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-envoy" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:02.368096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.368063 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-podinfo" pod "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" (UID: "6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:02.461529 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.461493 4981 scope.go:122] "RemoveContainer" containerID="7993f2e7dd705a8e79357c3086c11174540b7400b528528c830e4356d680059f" Nov 01 23:09:02.468584 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.468541 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.468584 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.468570 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.468760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.468599 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9rsb\" (UniqueName: \"kubernetes.io/projected/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-kube-api-access-k9rsb\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.468760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.468614 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:02.473276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.473252 4981 scope.go:122] "RemoveContainer" containerID="67b476b406db66566269c9e448ea859eca9eedbedab1c0d5c07320034eabb139" Nov 01 23:09:02.483941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.483924 4981 scope.go:122] "RemoveContainer" containerID="1d9c5a820cecdb7adfaf077d8bc3640c992ab3ff67c931c268403cd990e1114c" Nov 01 23:09:02.494142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.494121 4981 scope.go:122] "RemoveContainer" containerID="d8a5790fc877e94ceb6f5e1d9868da34777a1a2a6f83cb0306231c4f90d00600" Nov 01 23:09:02.504123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.504102 4981 scope.go:122] "RemoveContainer" containerID="c4db3e1058867c2a345a8173ebbe5ab99760039ae8f872ec0b581e113bb0073f" Nov 01 23:09:02.514315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.514298 4981 scope.go:122] "RemoveContainer" containerID="69ba2e04743cdf97c483a88633c92dc9424b07e253415785713e3ba7431486c1" Nov 01 23:09:02.524398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.524376 4981 scope.go:122] "RemoveContainer" containerID="d04fe4332f4d22b72c4b9c7e9af4f6369b1d88df94a03f7272b647ff17e376a8" Nov 01 23:09:02.646939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.646899 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" event={"ID":"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b","Type":"ContainerDied","Data":"69ba2e04743cdf97c483a88633c92dc9424b07e253415785713e3ba7431486c1"} Nov 01 23:09:02.647110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.646960 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" event={"ID":"6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b","Type":"ContainerDied","Data":"3e31280f38f70279a12ec42b3b042bfcd04f68a89529f284e414f2bdb62a7f46"} Nov 01 23:09:02.647110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.646989 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4" Nov 01 23:09:02.688793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.688715 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:02.689527 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:02.689498 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b1f5e_3ca2_4a32_bd38_724d06a1de6b.slice/crio-3e31280f38f70279a12ec42b3b042bfcd04f68a89529f284e414f2bdb62a7f46\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4b1f5e_3ca2_4a32_bd38_724d06a1de6b.slice\": RecentStats: unable to find data in memory cache]" Nov 01 23:09:02.689840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.689816 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4"] Nov 01 23:09:02.694609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.694562 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-49ab43-openshift-default-84f445c9b9-thmj4"] Nov 01 23:09:02.737163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:02.737082 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" path="/var/lib/kubelet/pods/6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b/volumes" Nov 01 23:09:03.035378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:03.035294 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb"] Nov 01 23:09:03.039893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:03.039864 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601a67c5-tpkmb"] Nov 01 23:09:03.488716 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:03.488684 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:04.156927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:04.156896 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:04.736288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:04.736260 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59177733-0923-4dfb-bec2-6004553b3c4e" path="/var/lib/kubelet/pods/59177733-0923-4dfb-bec2-6004553b3c4e/volumes" Nov 01 23:09:04.817919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:04.817872 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:05.475425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:05.475366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:05.575649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:05.575617 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:09:06.145327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:06.145294 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:06.584030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:06.584002 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:09:06.762663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:06.762600 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:07.496287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:07.496253 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:07.588793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:07.588751 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:08.251989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:08.251956 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:08.866755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:08.866711 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:09.708495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:09.708455 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:09.909197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:09.909145 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:10.061697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:10.061565 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6"] Nov 01 23:09:10.079828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:10.079786 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-53dd05-gh6k6"] Nov 01 23:09:10.115056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:10.115025 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:10.329342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:10.329253 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:10.550031 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:10.549995 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:10.737193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:10.737159 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b750fd83-0d02-40f4-9adf-fdfa3d885634" path="/var/lib/kubelet/pods/b750fd83-0d02-40f4-9adf-fdfa3d885634/volumes" Nov 01 23:09:11.000242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.000105 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:11.440034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.439991 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:11.536404 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.536368 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr"] Nov 01 23:09:11.536735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.536699 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" podUID="fdcc48c4-daf7-4243-9dc4-cc95ed523490" containerName="payload-pre-processing" containerID="cri-o://35c241da39ac7f38feac1dfa83fa8bda9352b24ebe20ca7777318edf72c1a28b" gracePeriod=30 Nov 01 23:09:11.556212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.556189 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:11.663532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.663502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:11.724660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.724255 4981 generic.go:356] "Generic (PLEG): container finished" podID="fdcc48c4-daf7-4243-9dc4-cc95ed523490" containerID="35c241da39ac7f38feac1dfa83fa8bda9352b24ebe20ca7777318edf72c1a28b" exitCode=0 Nov 01 23:09:11.724660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.724511 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" event={"ID":"fdcc48c4-daf7-4243-9dc4-cc95ed523490","Type":"ContainerDied","Data":"35c241da39ac7f38feac1dfa83fa8bda9352b24ebe20ca7777318edf72c1a28b"} Nov 01 23:09:11.874627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.830289 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:11.981772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:11.980928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:12.045250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.045218 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj"] Nov 01 23:09:12.051082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.051055 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-337dee-8swpj"] Nov 01 23:09:12.076113 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.076089 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:09:12.102124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.099323 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcc48c4-daf7-4243-9dc4-cc95ed523490-plugins-config-volume" pod "fdcc48c4-daf7-4243-9dc4-cc95ed523490" (UID: "fdcc48c4-daf7-4243-9dc4-cc95ed523490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:12.102124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.099804 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcc48c4-daf7-4243-9dc4-cc95ed523490-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcc48c4-daf7-4243-9dc4-cc95ed523490-plugins-config-volume\") pod \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " Nov 01 23:09:12.102124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.099944 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcc48c4-daf7-4243-9dc4-cc95ed523490-kube-api-access-m72ln\" (UniqueName: \"kubernetes.io/projected/fdcc48c4-daf7-4243-9dc4-cc95ed523490-kube-api-access-m72ln\") pod \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\" (UID: \"fdcc48c4-daf7-4243-9dc4-cc95ed523490\") " Nov 01 23:09:12.102124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.100707 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdcc48c4-daf7-4243-9dc4-cc95ed523490-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:12.120747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.113003 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcc48c4-daf7-4243-9dc4-cc95ed523490-kube-api-access-m72ln" pod "fdcc48c4-daf7-4243-9dc4-cc95ed523490" (UID: "fdcc48c4-daf7-4243-9dc4-cc95ed523490"). InnerVolumeSpecName "kube-api-access-m72ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:12.120747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.113990 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:12.202240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.202198 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m72ln\" (UniqueName: \"kubernetes.io/projected/fdcc48c4-daf7-4243-9dc4-cc95ed523490-kube-api-access-m72ln\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:12.455869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.455836 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:12.736842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.736744 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc07b2a4-972e-438a-a82c-0a4d617c169f" path="/var/lib/kubelet/pods/dc07b2a4-972e-438a-a82c-0a4d617c169f/volumes" Nov 01 23:09:12.736842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.736786 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" Nov 01 23:09:12.737489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.737444 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr" event={"ID":"fdcc48c4-daf7-4243-9dc4-cc95ed523490","Type":"ContainerDied","Data":"a250f0c38cb93333fd6b71d48a1a6b178fe3b26694b2b5851d93f3c6aafed610"} Nov 01 23:09:12.737653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.737496 4981 scope.go:122] "RemoveContainer" containerID="35c241da39ac7f38feac1dfa83fa8bda9352b24ebe20ca7777318edf72c1a28b" Nov 01 23:09:12.806312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.806278 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr"] Nov 01 23:09:12.821612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:12.821567 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-337dee-5748957695-2f9sr"] Nov 01 23:09:13.368636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.368600 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:13.431927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.429265 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj"] Nov 01 23:09:13.431927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.429500 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" podUID="86676654-36ab-49a6-bbb1-a9a6730d8fa7" containerName="payload-processing" containerID="cri-o://e3e668d1224fcaea8d4ddc304a1a1baa8532c5d3e7a9b81441dcd1d5f225b336" gracePeriod=30 Nov 01 23:09:13.440127 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.439867 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j"] Nov 01 23:09:13.440127 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.440098 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" podUID="3e3ee469-8880-43db-b734-6cecbb10f0e5" containerName="payload-pre-processing" containerID="cri-o://22741e46b8b2027a51f3ba6436a47913043f62ecba4d0b4057381bdb3d3e2d12" gracePeriod=30 Nov 01 23:09:13.481462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.481429 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:13.579764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.579412 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:13.729611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.717860 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:13.849754 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.849718 4981 generic.go:356] "Generic (PLEG): container finished" podID="3e3ee469-8880-43db-b734-6cecbb10f0e5" containerID="22741e46b8b2027a51f3ba6436a47913043f62ecba4d0b4057381bdb3d3e2d12" exitCode=0 Nov 01 23:09:13.850177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.850161 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" event={"ID":"3e3ee469-8880-43db-b734-6cecbb10f0e5","Type":"ContainerDied","Data":"22741e46b8b2027a51f3ba6436a47913043f62ecba4d0b4057381bdb3d3e2d12"} Nov 01 23:09:13.852382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.852364 4981 generic.go:356] "Generic (PLEG): container finished" podID="86676654-36ab-49a6-bbb1-a9a6730d8fa7" containerID="e3e668d1224fcaea8d4ddc304a1a1baa8532c5d3e7a9b81441dcd1d5f225b336" exitCode=0 Nov 01 23:09:13.852543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.852530 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" event={"ID":"86676654-36ab-49a6-bbb1-a9a6730d8fa7","Type":"ContainerDied","Data":"e3e668d1224fcaea8d4ddc304a1a1baa8532c5d3e7a9b81441dcd1d5f225b336"} Nov 01 23:09:13.857408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.857382 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:13.894829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.894803 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l"] Nov 01 23:09:13.903272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.903224 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9d2a4e-rvw9l"] Nov 01 23:09:13.920961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.920932 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:09:13.951869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:13.951841 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:14.024236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.024200 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86676654-36ab-49a6-bbb1-a9a6730d8fa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86676654-36ab-49a6-bbb1-a9a6730d8fa7-plugins-config-volume\") pod \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " Nov 01 23:09:14.024705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.024680 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86676654-36ab-49a6-bbb1-a9a6730d8fa7-kube-api-access-fjtjr\" (UniqueName: \"kubernetes.io/projected/86676654-36ab-49a6-bbb1-a9a6730d8fa7-kube-api-access-fjtjr\") pod \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\" (UID: \"86676654-36ab-49a6-bbb1-a9a6730d8fa7\") " Nov 01 23:09:14.026415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.026381 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86676654-36ab-49a6-bbb1-a9a6730d8fa7-plugins-config-volume" pod "86676654-36ab-49a6-bbb1-a9a6730d8fa7" (UID: "86676654-36ab-49a6-bbb1-a9a6730d8fa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:14.033119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.033024 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86676654-36ab-49a6-bbb1-a9a6730d8fa7-kube-api-access-fjtjr" pod "86676654-36ab-49a6-bbb1-a9a6730d8fa7" (UID: "86676654-36ab-49a6-bbb1-a9a6730d8fa7"). InnerVolumeSpecName "kube-api-access-fjtjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:14.042845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.042817 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:09:14.126880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.126848 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3ee469-8880-43db-b734-6cecbb10f0e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3ee469-8880-43db-b734-6cecbb10f0e5-plugins-config-volume\") pod \"3e3ee469-8880-43db-b734-6cecbb10f0e5\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " Nov 01 23:09:14.127082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.127002 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3ee469-8880-43db-b734-6cecbb10f0e5-kube-api-access-n5rjw\" (UniqueName: \"kubernetes.io/projected/3e3ee469-8880-43db-b734-6cecbb10f0e5-kube-api-access-n5rjw\") pod \"3e3ee469-8880-43db-b734-6cecbb10f0e5\" (UID: \"3e3ee469-8880-43db-b734-6cecbb10f0e5\") " Nov 01 23:09:14.127218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.127196 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3ee469-8880-43db-b734-6cecbb10f0e5-plugins-config-volume" pod "3e3ee469-8880-43db-b734-6cecbb10f0e5" (UID: "3e3ee469-8880-43db-b734-6cecbb10f0e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:14.127494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.127480 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3ee469-8880-43db-b734-6cecbb10f0e5-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.127537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.127498 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86676654-36ab-49a6-bbb1-a9a6730d8fa7-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.127537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.127507 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtjr\" (UniqueName: \"kubernetes.io/projected/86676654-36ab-49a6-bbb1-a9a6730d8fa7-kube-api-access-fjtjr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.129053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.129032 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3ee469-8880-43db-b734-6cecbb10f0e5-kube-api-access-n5rjw" pod "3e3ee469-8880-43db-b734-6cecbb10f0e5" (UID: "3e3ee469-8880-43db-b734-6cecbb10f0e5"). InnerVolumeSpecName "kube-api-access-n5rjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:14.229421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.229385 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rjw\" (UniqueName: \"kubernetes.io/projected/3e3ee469-8880-43db-b734-6cecbb10f0e5-kube-api-access-n5rjw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:14.347251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.347166 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:14.526963 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:14.526915 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:09:14.739724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.739678 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ab0689-7e04-4a3c-8b91-e07bd287833a" path="/var/lib/kubelet/pods/a5ab0689-7e04-4a3c-8b91-e07bd287833a/volumes" Nov 01 23:09:14.740451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.740426 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcc48c4-daf7-4243-9dc4-cc95ed523490" path="/var/lib/kubelet/pods/fdcc48c4-daf7-4243-9dc4-cc95ed523490/volumes" Nov 01 23:09:14.766939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.766910 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:14.867115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.867084 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" Nov 01 23:09:14.867115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.867105 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j" event={"ID":"3e3ee469-8880-43db-b734-6cecbb10f0e5","Type":"ContainerDied","Data":"29471182769fa1f05e3713b2cba534ff4166647fc244fb7733bd17991b7ec8c0"} Nov 01 23:09:14.867402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.867176 4981 scope.go:122] "RemoveContainer" containerID="22741e46b8b2027a51f3ba6436a47913043f62ecba4d0b4057381bdb3d3e2d12" Nov 01 23:09:14.869328 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.869301 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" event={"ID":"86676654-36ab-49a6-bbb1-a9a6730d8fa7","Type":"ContainerDied","Data":"ef93237e13ec550fcf9ef729e11f4f5ef9dcdeebc3624c58309230f442a78ee4"} Nov 01 23:09:14.869439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.869326 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj" Nov 01 23:09:14.880394 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.880243 4981 scope.go:122] "RemoveContainer" containerID="e3e668d1224fcaea8d4ddc304a1a1baa8532c5d3e7a9b81441dcd1d5f225b336" Nov 01 23:09:14.891999 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.891975 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j"] Nov 01 23:09:14.896236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.896213 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9d2a4e-f8f64b5f-8lj8j"] Nov 01 23:09:14.904841 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.904816 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj"] Nov 01 23:09:14.911896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:14.911876 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9d2a4e-666bf6dd58-wzzvj"] Nov 01 23:09:15.088884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:15.088808 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:15.274406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:15.274358 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:15.468142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:15.468114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:15.879842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:15.879750 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:16.247604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:16.247556 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:16.367740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:16.367708 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:16.736465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:16.736428 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3ee469-8880-43db-b734-6cecbb10f0e5" path="/var/lib/kubelet/pods/3e3ee469-8880-43db-b734-6cecbb10f0e5/volumes" Nov 01 23:09:16.736996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:16.736972 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86676654-36ab-49a6-bbb1-a9a6730d8fa7" path="/var/lib/kubelet/pods/86676654-36ab-49a6-bbb1-a9a6730d8fa7/volumes" Nov 01 23:09:16.986245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:16.986181 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:17.486419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:17.486391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:17.646216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:17.646181 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:17.749272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:17.749167 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:17.937485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:17.937451 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz"] Nov 01 23:09:17.937757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:17.937727 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" containerID="cri-o://5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00" gracePeriod=30 Nov 01 23:09:17.962724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:17.962692 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:18.357359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.357327 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:18.468855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.468824 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:18.611225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.611147 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8"] Nov 01 23:09:18.612003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.611983 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fdcc48c4-daf7-4243-9dc4-cc95ed523490" containerName="payload-pre-processing" Nov 01 23:09:18.612071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612004 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcc48c4-daf7-4243-9dc4-cc95ed523490" containerName="payload-pre-processing" Nov 01 23:09:18.612071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612025 4981 cpu_manager.go:414] "containerMap: removing container" podUID="86676654-36ab-49a6-bbb1-a9a6730d8fa7" containerName="payload-processing" Nov 01 23:09:18.612071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612030 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86676654-36ab-49a6-bbb1-a9a6730d8fa7" containerName="payload-processing" Nov 01 23:09:18.612071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612056 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" Nov 01 23:09:18.612071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612061 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612075 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3e3ee469-8880-43db-b734-6cecbb10f0e5" containerName="payload-pre-processing" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612081 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3ee469-8880-43db-b734-6cecbb10f0e5" containerName="payload-pre-processing" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612090 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612094 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612104 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b750fd83-0d02-40f4-9adf-fdfa3d885634" containerName="revoke-keys" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612108 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b750fd83-0d02-40f4-9adf-fdfa3d885634" containerName="revoke-keys" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612214 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612224 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b750fd83-0d02-40f4-9adf-fdfa3d885634" containerName="revoke-keys" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612234 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcc48c4-daf7-4243-9dc4-cc95ed523490" containerName="payload-pre-processing" Nov 01 23:09:18.612240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612243 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="86676654-36ab-49a6-bbb1-a9a6730d8fa7" containerName="payload-processing" Nov 01 23:09:18.612563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.612255 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3ee469-8880-43db-b734-6cecbb10f0e5" containerName="payload-pre-processing" Nov 01 23:09:18.617184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.617167 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.623800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.623775 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8"] Nov 01 23:09:18.710012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.709980 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:09:18.710174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.710128 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:09:18.770734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.770700 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/ba35e673-963e-4221-8503-2eb58e6ca473-kube-api-access-5xcmd\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.770959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.770937 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.771012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.770997 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.872620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.872496 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.872620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.872567 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.872871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.872663 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/ba35e673-963e-4221-8503-2eb58e6ca473-kube-api-access-5xcmd\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.873364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.873333 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.875119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.875096 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.881465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.881432 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/ba35e673-963e-4221-8503-2eb58e6ca473-kube-api-access-5xcmd\") pod \"maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:18.937169 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:18.937142 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:19.224679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:19.224644 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8"] Nov 01 23:09:19.224824 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:09:19.224712 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba35e673_963e_4221_8503_2eb58e6ca473.slice/crio-c39de4ceb00c7170382f1a48537522b51ee1cad07f403a260a3e69efe4e46b5b WatchSource:0}: Error finding container c39de4ceb00c7170382f1a48537522b51ee1cad07f403a260a3e69efe4e46b5b: Status 404 returned error can't find the container with id c39de4ceb00c7170382f1a48537522b51ee1cad07f403a260a3e69efe4e46b5b Nov 01 23:09:19.464938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:19.464904 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:19.927519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:19.927483 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" event={"ID":"ba35e673-963e-4221-8503-2eb58e6ca473","Type":"ContainerStarted","Data":"5ef9b45b0a61eb9cf0627c0d9cd4f4a4e30f8b8aa14db5427185dc7c3b9a2058"} Nov 01 23:09:19.927727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:19.927533 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" event={"ID":"ba35e673-963e-4221-8503-2eb58e6ca473","Type":"ContainerStarted","Data":"c39de4ceb00c7170382f1a48537522b51ee1cad07f403a260a3e69efe4e46b5b"} Nov 01 23:09:19.951837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:19.951788 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" podStartSLOduration=1.951773888 podStartE2EDuration="1.951773888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:19.948347987 +0000 UTC m=+2725.376225919" watchObservedRunningTime="2026-11-01 23:09:19.951773888 +0000 UTC m=+2725.379651819" Nov 01 23:09:20.162345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.162283 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:20.377637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.377540 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b"] Nov 01 23:09:20.379907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.378786 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4b1f5e-3ca2-4a32-bd38-724d06a1de6b" containerName="istio-proxy" Nov 01 23:09:20.384402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.384381 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.399354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.397326 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b"] Nov 01 23:09:20.473361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.473326 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-01e30d-openshift-default-dockercfg-vsndd\"" Nov 01 23:09:20.499312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499280 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499316 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499433 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499465 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqq6\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-kube-api-access-4hqq6\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499499 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499614 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499675 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499696 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.499920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.499897 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601433 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601391 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601465 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601492 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601538 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601585 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqq6\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-kube-api-access-4hqq6\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601619 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601669 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601737 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601765 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-credential-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.601962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.601947 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:20.602332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.602299 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-data\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.602403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.602337 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-socket\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.602470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.602443 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-certs\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.602895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.602870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istiod-ca-cert\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.604250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.604229 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-envoy\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.604423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.604406 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-podinfo\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.613836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.613806 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-token\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.613957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.613934 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqq6\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-kube-api-access-4hqq6\") pod \"e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.701699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.701658 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:20.717585 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.717536 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:20.879251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.879218 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k"] Nov 01 23:09:20.879483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.879431 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" containerID="cri-o://acb1e0898a35e354a8dd91006fae33c316f7e96c4aa4047c4b8f7c9b02dc1393" gracePeriod=30 Nov 01 23:09:20.903668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.903637 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:20.937138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.937107 4981 generic.go:356] "Generic (PLEG): container finished" podID="ba35e673-963e-4221-8503-2eb58e6ca473" containerID="5ef9b45b0a61eb9cf0627c0d9cd4f4a4e30f8b8aa14db5427185dc7c3b9a2058" exitCode=0 Nov 01 23:09:20.937301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:20.937196 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" event={"ID":"ba35e673-963e-4221-8503-2eb58e6ca473","Type":"ContainerDied","Data":"5ef9b45b0a61eb9cf0627c0d9cd4f4a4e30f8b8aa14db5427185dc7c3b9a2058"} Nov 01 23:09:21.009586 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.009487 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:21.027589 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.027551 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b"] Nov 01 23:09:21.027871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.027822 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:09:21.027998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.027978 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:09:21.028100 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.028081 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:09:21.386817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.386780 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb"] Nov 01 23:09:21.395094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.393083 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.398644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.398614 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb"] Nov 01 23:09:21.446014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.445978 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68gxz\" (UniqueName: \"kubernetes.io/projected/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-kube-api-access-68gxz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.446237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.446212 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.446330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.446255 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.547517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.547438 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gxz\" (UniqueName: \"kubernetes.io/projected/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-kube-api-access-68gxz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.547930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.547536 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.547930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.547559 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.548361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.548334 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.550096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.550071 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.556436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.556408 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gxz\" (UniqueName: \"kubernetes.io/projected/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-kube-api-access-68gxz\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.722078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.722050 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:21.947929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.947892 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" event={"ID":"39d5338e-6670-4c89-adbe-bb8850cbaaa4","Type":"ContainerStarted","Data":"30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a"} Nov 01 23:09:21.948115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:21.947957 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" event={"ID":"39d5338e-6670-4c89-adbe-bb8850cbaaa4","Type":"ContainerStarted","Data":"67e95b954cf9ac20478768fe820d7b1d8d09fcb27d0c12fc0be8aab88924e5c5"} Nov 01 23:09:22.007112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.007025 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" podStartSLOduration=2.00700431 podStartE2EDuration="2.00700431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:21.993174742 +0000 UTC m=+2727.421052674" watchObservedRunningTime="2026-11-01 23:09:22.00700431 +0000 UTC m=+2727.434882431" Nov 01 23:09:22.028302 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:09:22.028274 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5f4df4_2b48_46de_9e90_9e6fefb39a4d.slice/crio-a02ba85d8c9f204dce4ade1976a0dd3beb0c84a2145b4b57faf7ebd86f39919f WatchSource:0}: Error finding container a02ba85d8c9f204dce4ade1976a0dd3beb0c84a2145b4b57faf7ebd86f39919f: Status 404 returned error can't find the container with id a02ba85d8c9f204dce4ade1976a0dd3beb0c84a2145b4b57faf7ebd86f39919f Nov 01 23:09:22.029452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.029424 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb"] Nov 01 23:09:22.061639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.058744 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:22.215038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.214850 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:22.262333 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.262304 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-openshift-ca\") pod \"ba35e673-963e-4221-8503-2eb58e6ca473\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " Nov 01 23:09:22.262533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.262493 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-xks-ca\") pod \"ba35e673-963e-4221-8503-2eb58e6ca473\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " Nov 01 23:09:22.262652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.262602 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba35e673-963e-4221-8503-2eb58e6ca473-kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/ba35e673-963e-4221-8503-2eb58e6ca473-kube-api-access-5xcmd\") pod \"ba35e673-963e-4221-8503-2eb58e6ca473\" (UID: \"ba35e673-963e-4221-8503-2eb58e6ca473\") " Nov 01 23:09:22.262821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.262774 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-openshift-ca" pod "ba35e673-963e-4221-8503-2eb58e6ca473" (UID: "ba35e673-963e-4221-8503-2eb58e6ca473"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:22.263239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.263214 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:22.265289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.265263 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba35e673-963e-4221-8503-2eb58e6ca473-kube-api-access-5xcmd" pod "ba35e673-963e-4221-8503-2eb58e6ca473" (UID: "ba35e673-963e-4221-8503-2eb58e6ca473"). InnerVolumeSpecName "kube-api-access-5xcmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:22.265403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.265287 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-xks-ca" pod "ba35e673-963e-4221-8503-2eb58e6ca473" (UID: "ba35e673-963e-4221-8503-2eb58e6ca473"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:22.364257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.364213 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xcmd\" (UniqueName: \"kubernetes.io/projected/ba35e673-963e-4221-8503-2eb58e6ca473-kube-api-access-5xcmd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:22.364257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.364256 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba35e673-963e-4221-8503-2eb58e6ca473-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:22.717753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.717720 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:22.719241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.719217 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 01 23:09:22.719383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.719273 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 01 23:09:22.960342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.960310 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" event={"ID":"da5f4df4-2b48-46de-9e90-9e6fefb39a4d","Type":"ContainerStarted","Data":"92952219c6f87c63822cf937302b5e7912cefaa4eb12be557a6d2b7be0f5c884"} Nov 01 23:09:22.960492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.960373 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" event={"ID":"da5f4df4-2b48-46de-9e90-9e6fefb39a4d","Type":"ContainerStarted","Data":"a02ba85d8c9f204dce4ade1976a0dd3beb0c84a2145b4b57faf7ebd86f39919f"} Nov 01 23:09:22.961938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.961908 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" event={"ID":"ba35e673-963e-4221-8503-2eb58e6ca473","Type":"ContainerDied","Data":"c39de4ceb00c7170382f1a48537522b51ee1cad07f403a260a3e69efe4e46b5b"} Nov 01 23:09:22.961938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.961933 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8" Nov 01 23:09:22.962077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:22.961961 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c39de4ceb00c7170382f1a48537522b51ee1cad07f403a260a3e69efe4e46b5b" Nov 01 23:09:23.008941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.008888 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" podStartSLOduration=2.008873849 podStartE2EDuration="2.008873849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:22.995025989 +0000 UTC m=+2728.422903920" watchObservedRunningTime="2026-11-01 23:09:23.008873849 +0000 UTC m=+2728.436751779" Nov 01 23:09:23.339272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.339241 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz"] Nov 01 23:09:23.366354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.366292 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:23.371785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.371760 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:09:23.401086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.375407 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb"] Nov 01 23:09:23.513658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.506968 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526092 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-kube-api-access-tmtt4\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-kube-api-access-tmtt4\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526269 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1eff244-27b3-4175-8e14-b812d92860a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1eff244-27b3-4175-8e14-b812d92860a9-istiod-ca-cert\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526312 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-istio-token\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-istio-token\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526415 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-certs\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526452 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-credential-socket\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526538 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-envoy\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526590 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-socket\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526661 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1eff244-27b3-4175-8e14-b812d92860a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1eff244-27b3-4175-8e14-b812d92860a9-istio-podinfo\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.526827 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-data\") pod \"a1eff244-27b3-4175-8e14-b812d92860a9\" (UID: \"a1eff244-27b3-4175-8e14-b812d92860a9\") " Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.528182 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-certs" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.528600 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-data" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.528837 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-credential-socket" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.529311 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-socket" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.531598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.529619 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1eff244-27b3-4175-8e14-b812d92860a9-istiod-ca-cert" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:23.552177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.551656 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-envoy" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:23.552177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.552127 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-istio-token" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:23.552453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.552240 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-kube-api-access-tmtt4" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "kube-api-access-tmtt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:23.553614 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.553588 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1eff244-27b3-4175-8e14-b812d92860a9-istio-podinfo" pod "a1eff244-27b3-4175-8e14-b812d92860a9" (UID: "a1eff244-27b3-4175-8e14-b812d92860a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:23.629757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629723 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.629757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629751 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.629757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629760 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.630017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629769 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1eff244-27b3-4175-8e14-b812d92860a9-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.630017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629778 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.630017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629788 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmtt4\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-kube-api-access-tmtt4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.630017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629798 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1eff244-27b3-4175-8e14-b812d92860a9-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.630017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629807 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1eff244-27b3-4175-8e14-b812d92860a9-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.630017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.629815 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1eff244-27b3-4175-8e14-b812d92860a9-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:23.718874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.718834 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 01 23:09:23.719355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.718883 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 01 23:09:23.971773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.971741 4981 generic.go:356] "Generic (PLEG): container finished" podID="da5f4df4-2b48-46de-9e90-9e6fefb39a4d" containerID="92952219c6f87c63822cf937302b5e7912cefaa4eb12be557a6d2b7be0f5c884" exitCode=0 Nov 01 23:09:23.971968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.971834 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" event={"ID":"da5f4df4-2b48-46de-9e90-9e6fefb39a4d","Type":"ContainerDied","Data":"92952219c6f87c63822cf937302b5e7912cefaa4eb12be557a6d2b7be0f5c884"} Nov 01 23:09:23.973193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.973171 4981 generic.go:356] "Generic (PLEG): container finished" podID="a1eff244-27b3-4175-8e14-b812d92860a9" containerID="5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00" exitCode=0 Nov 01 23:09:23.973304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.973200 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" event={"ID":"a1eff244-27b3-4175-8e14-b812d92860a9","Type":"ContainerDied","Data":"5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00"} Nov 01 23:09:23.973304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.973216 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" Nov 01 23:09:23.973304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.973257 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz" event={"ID":"a1eff244-27b3-4175-8e14-b812d92860a9","Type":"ContainerDied","Data":"be80fc3d72a8c66ce509c4b8c242d35009b8589e9e4a973b300a8b27c538c67a"} Nov 01 23:09:23.973304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:23.973292 4981 scope.go:122] "RemoveContainer" containerID="5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00" Nov 01 23:09:24.019360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.018093 4981 scope.go:122] "RemoveContainer" containerID="5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00" Nov 01 23:09:24.019360 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:24.018592 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00\": container with ID starting with 5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00 not found: ID does not exist" containerID="5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00" Nov 01 23:09:24.019360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.018630 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00"} err="failed to get container status \"5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00\": rpc error: code = NotFound desc = could not find container \"5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00\": container with ID starting with 5c7fffa392a956129d4f34deb8008239d114a6cdba69fe6392e608cb32f02d00 not found: ID does not exist" Nov 01 23:09:24.034721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.034687 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz"] Nov 01 23:09:24.039263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.039237 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-53dd05-openshift-default-76984d9b67-cqdqz"] Nov 01 23:09:24.298645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.295617 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:24.717892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.717854 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 01 23:09:24.718090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.717906 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 01 23:09:24.737550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:24.737511 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" path="/var/lib/kubelet/pods/a1eff244-27b3-4175-8e14-b812d92860a9/volumes" Nov 01 23:09:25.107315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.107033 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:25.254242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.254213 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:25.425675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.425646 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-kube-api-access-68gxz\" (UniqueName: \"kubernetes.io/projected/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-kube-api-access-68gxz\") pod \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " Nov 01 23:09:25.425854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.425738 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-xks-ca\") pod \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " Nov 01 23:09:25.425900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.425889 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-openshift-ca\") pod \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\" (UID: \"da5f4df4-2b48-46de-9e90-9e6fefb39a4d\") " Nov 01 23:09:25.426293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.426256 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-openshift-ca" pod "da5f4df4-2b48-46de-9e90-9e6fefb39a4d" (UID: "da5f4df4-2b48-46de-9e90-9e6fefb39a4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:25.426483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.426465 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:25.427921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.427894 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-kube-api-access-68gxz" pod "da5f4df4-2b48-46de-9e90-9e6fefb39a4d" (UID: "da5f4df4-2b48-46de-9e90-9e6fefb39a4d"). InnerVolumeSpecName "kube-api-access-68gxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:25.428020 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.427944 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-xks-ca" pod "da5f4df4-2b48-46de-9e90-9e6fefb39a4d" (UID: "da5f4df4-2b48-46de-9e90-9e6fefb39a4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:25.527472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.527433 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68gxz\" (UniqueName: \"kubernetes.io/projected/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-kube-api-access-68gxz\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:25.527472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.527467 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5f4df4-2b48-46de-9e90-9e6fefb39a4d-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:25.717866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.717777 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 01 23:09:25.717866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:25.717833 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 01 23:09:26.003257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.003146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:26.017648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.017611 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" event={"ID":"da5f4df4-2b48-46de-9e90-9e6fefb39a4d","Type":"ContainerDied","Data":"a02ba85d8c9f204dce4ade1976a0dd3beb0c84a2145b4b57faf7ebd86f39919f"} Nov 01 23:09:26.017648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.017638 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb" Nov 01 23:09:26.017884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.017677 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a02ba85d8c9f204dce4ade1976a0dd3beb0c84a2145b4b57faf7ebd86f39919f" Nov 01 23:09:26.020007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.019977 4981 generic.go:356] "Generic (PLEG): container finished" podID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerID="acb1e0898a35e354a8dd91006fae33c316f7e96c4aa4047c4b8f7c9b02dc1393" exitCode=0 Nov 01 23:09:26.020165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.020111 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" event={"ID":"560eacaa-6c50-4c64-8650-023dc6c924a6","Type":"ContainerDied","Data":"acb1e0898a35e354a8dd91006fae33c316f7e96c4aa4047c4b8f7c9b02dc1393"} Nov 01 23:09:26.269854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.269787 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:09:26.456930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.456901 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-kube-api-access-7c4tl\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-kube-api-access-7c4tl\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.456966 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-podinfo\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.456990 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-data\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457049 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-token\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-token\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457107 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-socket\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457237 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-certs\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457332 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-data" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.457373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457362 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-credential-socket\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457371 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-socket" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.457554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457502 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-envoy\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457536 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/560eacaa-6c50-4c64-8650-023dc6c924a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560eacaa-6c50-4c64-8650-023dc6c924a6-istiod-ca-cert\") pod \"560eacaa-6c50-4c64-8650-023dc6c924a6\" (UID: \"560eacaa-6c50-4c64-8650-023dc6c924a6\") " Nov 01 23:09:26.457705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457545 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-certs" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.457705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.457652 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-credential-socket" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.458102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.458048 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560eacaa-6c50-4c64-8650-023dc6c924a6-istiod-ca-cert" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:26.458234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.458136 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.458234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.458158 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.458234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.458173 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.458234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.458187 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.459042 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.459020 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-kube-api-access-7c4tl" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "kube-api-access-7c4tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.459314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.459296 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-podinfo" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:26.459622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.459601 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-envoy" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:26.459729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.459692 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-token" pod "560eacaa-6c50-4c64-8650-023dc6c924a6" (UID: "560eacaa-6c50-4c64-8650-023dc6c924a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:26.560143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.560015 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.560143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.560048 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/560eacaa-6c50-4c64-8650-023dc6c924a6-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.560143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.560064 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4tl\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-kube-api-access-7c4tl\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.560143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.560079 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.560143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.560091 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/560eacaa-6c50-4c64-8650-023dc6c924a6-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:26.718017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.717985 4981 patch_prober.go:29] interesting pod/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 01 23:09:26.718284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.718043 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 01 23:09:26.895512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:26.895421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:27.029909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.029876 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" Nov 01 23:09:27.029909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.029892 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k" event={"ID":"560eacaa-6c50-4c64-8650-023dc6c924a6","Type":"ContainerDied","Data":"633ba195a582683c8a803d73c397ef0567f1d3f1bbb78bea3bcc25676a12a143"} Nov 01 23:09:27.030463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.029966 4981 scope.go:122] "RemoveContainer" containerID="acb1e0898a35e354a8dd91006fae33c316f7e96c4aa4047c4b8f7c9b02dc1393" Nov 01 23:09:27.054416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.054383 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k"] Nov 01 23:09:27.065882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.065856 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9d2a4e-openshift-default-64594646dc-w7q9k"] Nov 01 23:09:27.646058 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.646023 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:27.723397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.723366 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:27.723769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.723745 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:27.724554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.724535 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:09:27.750355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.750319 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:27.847825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:27.847792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:28.538296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:28.538211 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:28.739534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:28.739502 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" path="/var/lib/kubelet/pods/560eacaa-6c50-4c64-8650-023dc6c924a6/volumes" Nov 01 23:09:28.821056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:28.820972 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:28.919653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:28.919618 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:29.577542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:29.577509 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:30.252211 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:30.252182 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:30.929753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:30.929722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:31.053835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:31.053800 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:31.606218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:31.606180 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6"] Nov 01 23:09:31.701812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:31.701780 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:31.794661 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:31.794630 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:31.967624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:31.967567 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:32.867675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:32.867640 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:33.678826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:33.678796 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:34.359451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:34.359418 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:35.027624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:35.027591 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:35.669638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:35.669561 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:36.265547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:36.265513 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:36.607534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:36.607431 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" podUID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" containerName="payload-processing" containerID="cri-o://16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b" gracePeriod=25 Nov 01 23:09:36.889624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:36.889518 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:37.570263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:37.570218 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:38.212502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.212463 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:38.364126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.364091 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" containerID="cri-o://786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3" gracePeriod=45 Nov 01 23:09:38.430345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.430305 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" containerID="cri-o://18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009" gracePeriod=45 Nov 01 23:09:38.750382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.750357 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 23:09:38.754037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.754016 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 23:09:38.877100 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.877021 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.938840 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-model-cache\") pod \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.938939 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-dshm\") pod \"42e7221b-ee32-4b06-854e-b90394314b45\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939004 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-kube-api-access-td8dx\" (UniqueName: \"kubernetes.io/projected/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-kube-api-access-td8dx\") pod \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939165 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-home\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-home\") pod \"42e7221b-ee32-4b06-854e-b90394314b45\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939238 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-dshm\") pod \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939284 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42e7221b-ee32-4b06-854e-b90394314b45-tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7221b-ee32-4b06-854e-b90394314b45-tls-certs\") pod \"42e7221b-ee32-4b06-854e-b90394314b45\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939378 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tls-certs\") pod \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939437 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-home\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-home\") pod \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939463 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e7221b-ee32-4b06-854e-b90394314b45-kube-api-access-sr6d7\" (UniqueName: \"kubernetes.io/projected/42e7221b-ee32-4b06-854e-b90394314b45-kube-api-access-sr6d7\") pod \"42e7221b-ee32-4b06-854e-b90394314b45\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939553 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-tmp-dir\") pod \"42e7221b-ee32-4b06-854e-b90394314b45\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939611 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-model-cache\") pod \"42e7221b-ee32-4b06-854e-b90394314b45\" (UID: \"42e7221b-ee32-4b06-854e-b90394314b45\") " Nov 01 23:09:38.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.939671 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tmp-dir\") pod \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\" (UID: \"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd\") " Nov 01 23:09:38.943558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.943516 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tmp-dir" pod "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" (UID: "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:38.949629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.947985 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-home" pod "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" (UID: "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:38.949800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.949656 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-tmp-dir" pod "42e7221b-ee32-4b06-854e-b90394314b45" (UID: "42e7221b-ee32-4b06-854e-b90394314b45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:38.949873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.949854 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-model-cache" pod "42e7221b-ee32-4b06-854e-b90394314b45" (UID: "42e7221b-ee32-4b06-854e-b90394314b45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:38.950962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.950609 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-home" pod "42e7221b-ee32-4b06-854e-b90394314b45" (UID: "42e7221b-ee32-4b06-854e-b90394314b45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:38.967336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.952814 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-model-cache" pod "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" (UID: "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:38.967336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.955375 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tls-certs" pod "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" (UID: "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:38.967336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.960762 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-kube-api-access-td8dx" pod "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" (UID: "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd"). InnerVolumeSpecName "kube-api-access-td8dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:38.967759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.967708 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-dshm" pod "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" (UID: "ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:38.968447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.968403 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e7221b-ee32-4b06-854e-b90394314b45-kube-api-access-sr6d7" pod "42e7221b-ee32-4b06-854e-b90394314b45" (UID: "42e7221b-ee32-4b06-854e-b90394314b45"). InnerVolumeSpecName "kube-api-access-sr6d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:38.969465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.969444 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e7221b-ee32-4b06-854e-b90394314b45-tls-certs" pod "42e7221b-ee32-4b06-854e-b90394314b45" (UID: "42e7221b-ee32-4b06-854e-b90394314b45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:38.970849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:38.970759 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-dshm" pod "42e7221b-ee32-4b06-854e-b90394314b45" (UID: "42e7221b-ee32-4b06-854e-b90394314b45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:39.041826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041790 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.041826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041822 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.041826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041834 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42e7221b-ee32-4b06-854e-b90394314b45-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041854 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041866 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041878 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr6d7\" (UniqueName: \"kubernetes.io/projected/42e7221b-ee32-4b06-854e-b90394314b45-kube-api-access-sr6d7\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041889 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041902 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041913 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041925 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041937 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42e7221b-ee32-4b06-854e-b90394314b45-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.042101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.041952 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td8dx\" (UniqueName: \"kubernetes.io/projected/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd-kube-api-access-td8dx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:39.178009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.177975 4981 generic.go:356] "Generic (PLEG): container finished" podID="42e7221b-ee32-4b06-854e-b90394314b45" containerID="786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3" exitCode=0 Nov 01 23:09:39.178185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.178020 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" Nov 01 23:09:39.178185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.178057 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" event={"ID":"42e7221b-ee32-4b06-854e-b90394314b45","Type":"ContainerDied","Data":"786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3"} Nov 01 23:09:39.178185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.178107 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz" event={"ID":"42e7221b-ee32-4b06-854e-b90394314b45","Type":"ContainerDied","Data":"6618a76aef159a2066167066390470eb70c659262c8dff14c19d9ea0f8fe4f61"} Nov 01 23:09:39.178185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.178126 4981 scope.go:122] "RemoveContainer" containerID="786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3" Nov 01 23:09:39.179802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.179772 4981 generic.go:356] "Generic (PLEG): container finished" podID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerID="18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009" exitCode=0 Nov 01 23:09:39.179929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.179801 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" event={"ID":"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd","Type":"ContainerDied","Data":"18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009"} Nov 01 23:09:39.179929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.179838 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" event={"ID":"ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd","Type":"ContainerDied","Data":"5c97a5597cf5e047dfecb1e3050db8b6a461565fd47df109df049183e9ae7a80"} Nov 01 23:09:39.179929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.179861 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb" Nov 01 23:09:39.192681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.192665 4981 scope.go:122] "RemoveContainer" containerID="786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3" Nov 01 23:09:39.192923 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:39.192905 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3\": container with ID starting with 786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3 not found: ID does not exist" containerID="786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3" Nov 01 23:09:39.192988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.192930 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3"} err="failed to get container status \"786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3\": rpc error: code = NotFound desc = could not find container \"786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3\": container with ID starting with 786ce1425b845eb55fad71181d27eaf6914f6faa2e16242c3f618d4e09a05fc3 not found: ID does not exist" Nov 01 23:09:39.192988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.192946 4981 scope.go:122] "RemoveContainer" containerID="18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009" Nov 01 23:09:39.203699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.203684 4981 scope.go:122] "RemoveContainer" containerID="18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009" Nov 01 23:09:39.204034 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:39.204008 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009\": container with ID starting with 18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009 not found: ID does not exist" containerID="18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009" Nov 01 23:09:39.204144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.204039 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009"} err="failed to get container status \"18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009\": rpc error: code = NotFound desc = could not find container \"18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009\": container with ID starting with 18a47c0ea9dd62f39307f1c4b7f22bb1eee3c56908c65c4846702305e9b52009 not found: ID does not exist" Nov 01 23:09:39.208341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.208323 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz"] Nov 01 23:09:39.215253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.215232 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/distinct-2-w6-927b68-kserve-68d6fcfd64-d85vz"] Nov 01 23:09:39.222016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.221993 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb"] Nov 01 23:09:39.225531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.225515 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-927b68/simulator-w6-927b68-kserve-69787658df-mmvbb"] Nov 01 23:09:39.431560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.431483 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:39.618598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.614229 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:39.738807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.738706 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:39.814014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.813981 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb"] Nov 01 23:09:39.818459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:39.818429 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-788d7b-v9gsb"] Nov 01 23:09:40.059993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.059868 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:40.222123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.222094 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:40.647430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.647397 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx"] Nov 01 23:09:40.648258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648233 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" Nov 01 23:09:40.648258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648258 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648275 4981 cpu_manager.go:414] "containerMap: removing container" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648283 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648315 4981 cpu_manager.go:414] "containerMap: removing container" podUID="da5f4df4-2b48-46de-9e90-9e6fefb39a4d" containerName="revoke-keys" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648323 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5f4df4-2b48-46de-9e90-9e6fefb39a4d" containerName="revoke-keys" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648341 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648347 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648358 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ba35e673-963e-4221-8503-2eb58e6ca473" containerName="revoke-keys" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648365 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba35e673-963e-4221-8503-2eb58e6ca473" containerName="revoke-keys" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648396 4981 cpu_manager.go:414] "containerMap: removing container" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" Nov 01 23:09:40.648447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648403 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" Nov 01 23:09:40.649011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648593 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1eff244-27b3-4175-8e14-b812d92860a9" containerName="istio-proxy" Nov 01 23:09:40.649011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648608 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="560eacaa-6c50-4c64-8650-023dc6c924a6" containerName="istio-proxy" Nov 01 23:09:40.649011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648622 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5f4df4-2b48-46de-9e90-9e6fefb39a4d" containerName="revoke-keys" Nov 01 23:09:40.649011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648634 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" containerName="main" Nov 01 23:09:40.649011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648645 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba35e673-963e-4221-8503-2eb58e6ca473" containerName="revoke-keys" Nov 01 23:09:40.649011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.648660 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e7221b-ee32-4b06-854e-b90394314b45" containerName="main" Nov 01 23:09:40.655934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.655494 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.668717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.668681 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80621398-2493-40fc-a370-001e18141226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.668945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.668920 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80621398-2493-40fc-a370-001e18141226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.669224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.669198 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6xj\" (UniqueName: \"kubernetes.io/projected/80621398-2493-40fc-a370-001e18141226-kube-api-access-vd6xj\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.670836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.669720 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z"] Nov 01 23:09:40.681359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.680373 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.685121 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.685098 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx"] Nov 01 23:09:40.692611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.692565 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z"] Nov 01 23:09:40.702269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.702245 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:09:40.702399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.702339 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:09:40.740082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.740050 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e7221b-ee32-4b06-854e-b90394314b45" path="/var/lib/kubelet/pods/42e7221b-ee32-4b06-854e-b90394314b45/volumes" Nov 01 23:09:40.740562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.740545 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd" path="/var/lib/kubelet/pods/ce6fd62c-f978-4fd8-87f9-cc25c5b8b7cd/volumes" Nov 01 23:09:40.741012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.740999 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5f4df4-2b48-46de-9e90-9e6fefb39a4d" path="/var/lib/kubelet/pods/da5f4df4-2b48-46de-9e90-9e6fefb39a4d/volumes" Nov 01 23:09:40.772279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.772247 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.772438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.772291 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80621398-2493-40fc-a370-001e18141226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.772438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.772380 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6xj\" (UniqueName: \"kubernetes.io/projected/80621398-2493-40fc-a370-001e18141226-kube-api-access-vd6xj\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.772438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.772424 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jql\" (UniqueName: \"kubernetes.io/projected/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-kube-api-access-b7jql\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.772693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.772534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.772693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.772647 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80621398-2493-40fc-a370-001e18141226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.773237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.773218 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80621398-2493-40fc-a370-001e18141226-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.774927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.774862 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80621398-2493-40fc-a370-001e18141226-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.785869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.785838 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6xj\" (UniqueName: \"kubernetes.io/projected/80621398-2493-40fc-a370-001e18141226-kube-api-access-vd6xj\") pod \"maas-api-revoke-sub-3d6ddc91-7zzjx\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:40.873979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.873932 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.874484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.874029 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jql\" (UniqueName: \"kubernetes.io/projected/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-kube-api-access-b7jql\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.874484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.874134 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.874770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.874745 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.876448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.876420 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.885723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.885702 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jql\" (UniqueName: \"kubernetes.io/projected/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-kube-api-access-b7jql\") pod \"maas-api-revoke-sub-0e0862ce-ssd2z\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:40.939173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.939147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:40.988447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:40.988408 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:41.009933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:41.009897 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:41.319133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:41.319093 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx"] Nov 01 23:09:41.321204 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:09:41.321180 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80621398_2493_40fc_a370_001e18141226.slice/crio-90cd7b24d4dbaa65fdd910123500f91f605a764bed129ba59136d69f0fe4d1a3 WatchSource:0}: Error finding container 90cd7b24d4dbaa65fdd910123500f91f605a764bed129ba59136d69f0fe4d1a3: Status 404 returned error can't find the container with id 90cd7b24d4dbaa65fdd910123500f91f605a764bed129ba59136d69f0fe4d1a3 Nov 01 23:09:41.322799 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:09:41.322775 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7f5f55_b28d_4e63_8c30_36bb2ff39c99.slice/crio-143f0d90ca6ed921275be5815557e72e5416ee27d1402dd4297a9dcb4258c153 WatchSource:0}: Error finding container 143f0d90ca6ed921275be5815557e72e5416ee27d1402dd4297a9dcb4258c153: Status 404 returned error can't find the container with id 143f0d90ca6ed921275be5815557e72e5416ee27d1402dd4297a9dcb4258c153 Nov 01 23:09:41.324649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:41.324623 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z"] Nov 01 23:09:41.444443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:41.444411 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" podUID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:41.630753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:41.630673 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:42.213182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:42.213135 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" event={"ID":"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99","Type":"ContainerStarted","Data":"63d0196cb68bc6b9a7654dbc82996c33918de8682b51f2f2aca000e3a1ab3e92"} Nov 01 23:09:42.213678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:42.213202 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" event={"ID":"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99","Type":"ContainerStarted","Data":"143f0d90ca6ed921275be5815557e72e5416ee27d1402dd4297a9dcb4258c153"} Nov 01 23:09:42.215158 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:42.215125 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" event={"ID":"80621398-2493-40fc-a370-001e18141226","Type":"ContainerStarted","Data":"fc746415e9feaf6ab85780d33832c9cd93e06b95c6e50ab603026f14303d0257"} Nov 01 23:09:42.215256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:42.215172 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" event={"ID":"80621398-2493-40fc-a370-001e18141226","Type":"ContainerStarted","Data":"90cd7b24d4dbaa65fdd910123500f91f605a764bed129ba59136d69f0fe4d1a3"} Nov 01 23:09:42.231701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:42.231661 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" podStartSLOduration=2.231647672 podStartE2EDuration="2.231647672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:42.229151203 +0000 UTC m=+2747.657029133" watchObservedRunningTime="2026-11-01 23:09:42.231647672 +0000 UTC m=+2747.659525594" Nov 01 23:09:42.249717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:42.249652 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" podStartSLOduration=2.249636461 podStartE2EDuration="2.249636461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:42.245889907 +0000 UTC m=+2747.673767839" watchObservedRunningTime="2026-11-01 23:09:42.249636461 +0000 UTC m=+2747.677514393" Nov 01 23:09:42.282797 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:42.282775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:43.089762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:43.089731 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:43.225457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:43.225423 4981 generic.go:356] "Generic (PLEG): container finished" podID="80621398-2493-40fc-a370-001e18141226" containerID="fc746415e9feaf6ab85780d33832c9cd93e06b95c6e50ab603026f14303d0257" exitCode=0 Nov 01 23:09:43.225953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:43.225503 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" event={"ID":"80621398-2493-40fc-a370-001e18141226","Type":"ContainerDied","Data":"fc746415e9feaf6ab85780d33832c9cd93e06b95c6e50ab603026f14303d0257"} Nov 01 23:09:43.227681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:43.227657 4981 generic.go:356] "Generic (PLEG): container finished" podID="bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" containerID="63d0196cb68bc6b9a7654dbc82996c33918de8682b51f2f2aca000e3a1ab3e92" exitCode=0 Nov 01 23:09:43.227790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:43.227692 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" event={"ID":"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99","Type":"ContainerDied","Data":"63d0196cb68bc6b9a7654dbc82996c33918de8682b51f2f2aca000e3a1ab3e92"} Nov 01 23:09:43.745056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:43.745023 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:44.411851 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.411818 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:44.429741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.429244 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-kube-api-access-b7jql\" (UniqueName: \"kubernetes.io/projected/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-kube-api-access-b7jql\") pod \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " Nov 01 23:09:44.429741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.429320 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-xks-ca\") pod \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " Nov 01 23:09:44.429741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.429525 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-openshift-ca\") pod \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\" (UID: \"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99\") " Nov 01 23:09:44.431451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.431151 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-openshift-ca" pod "bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" (UID: "bd7f5f55-b28d-4e63-8c30-36bb2ff39c99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:44.443429 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.443401 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-xks-ca" pod "bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" (UID: "bd7f5f55-b28d-4e63-8c30-36bb2ff39c99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:44.443521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.443424 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-kube-api-access-b7jql" pod "bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" (UID: "bd7f5f55-b28d-4e63-8c30-36bb2ff39c99"). InnerVolumeSpecName "kube-api-access-b7jql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:44.472442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.472418 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:44.487074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.487015 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:44.562659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.556991 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:44.562659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.557874 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jql\" (UniqueName: \"kubernetes.io/projected/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-kube-api-access-b7jql\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:44.562659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.557900 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:44.562659 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:44.560933 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:09:44.659214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.659181 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80621398-2493-40fc-a370-001e18141226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80621398-2493-40fc-a370-001e18141226-maas-api-openshift-ca\") pod \"80621398-2493-40fc-a370-001e18141226\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " Nov 01 23:09:44.659396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.659340 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80621398-2493-40fc-a370-001e18141226-kube-api-access-vd6xj\" (UniqueName: \"kubernetes.io/projected/80621398-2493-40fc-a370-001e18141226-kube-api-access-vd6xj\") pod \"80621398-2493-40fc-a370-001e18141226\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " Nov 01 23:09:44.659533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.659515 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80621398-2493-40fc-a370-001e18141226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80621398-2493-40fc-a370-001e18141226-maas-api-xks-ca\") pod \"80621398-2493-40fc-a370-001e18141226\" (UID: \"80621398-2493-40fc-a370-001e18141226\") " Nov 01 23:09:44.659626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.659569 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80621398-2493-40fc-a370-001e18141226-maas-api-openshift-ca" pod "80621398-2493-40fc-a370-001e18141226" (UID: "80621398-2493-40fc-a370-001e18141226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:44.660119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.660092 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80621398-2493-40fc-a370-001e18141226-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:44.661420 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.661396 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80621398-2493-40fc-a370-001e18141226-kube-api-access-vd6xj" pod "80621398-2493-40fc-a370-001e18141226" (UID: "80621398-2493-40fc-a370-001e18141226"). InnerVolumeSpecName "kube-api-access-vd6xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:44.661522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.661425 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80621398-2493-40fc-a370-001e18141226-maas-api-xks-ca" pod "80621398-2493-40fc-a370-001e18141226" (UID: "80621398-2493-40fc-a370-001e18141226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:44.761288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.761256 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6xj\" (UniqueName: \"kubernetes.io/projected/80621398-2493-40fc-a370-001e18141226-kube-api-access-vd6xj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:44.761288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:44.761284 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80621398-2493-40fc-a370-001e18141226-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:45.245482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:45.245452 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" Nov 01 23:09:45.245482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:45.245463 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z" event={"ID":"bd7f5f55-b28d-4e63-8c30-36bb2ff39c99","Type":"ContainerDied","Data":"143f0d90ca6ed921275be5815557e72e5416ee27d1402dd4297a9dcb4258c153"} Nov 01 23:09:45.245850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:45.245545 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143f0d90ca6ed921275be5815557e72e5416ee27d1402dd4297a9dcb4258c153" Nov 01 23:09:45.248809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:45.247551 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" event={"ID":"80621398-2493-40fc-a370-001e18141226","Type":"ContainerDied","Data":"90cd7b24d4dbaa65fdd910123500f91f605a764bed129ba59136d69f0fe4d1a3"} Nov 01 23:09:45.248809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:45.247636 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90cd7b24d4dbaa65fdd910123500f91f605a764bed129ba59136d69f0fe4d1a3" Nov 01 23:09:45.248809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:45.247591 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx" Nov 01 23:09:45.463521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:45.463491 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:46.099247 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:46.099216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:46.864163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:46.864127 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:46.978171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:46.976692 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:47.148207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.148118 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp"] Nov 01 23:09:47.148722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.148688 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" containerID="cri-o://e7be2de8a8ec2c64cb8f3dbaad546b12cbeb946a1b2affd1cb538bd9ba3dc4e8" gracePeriod=30 Nov 01 23:09:47.207319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.207291 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:47.644826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.644793 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v"] Nov 01 23:09:47.645521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.645507 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" containerName="revoke-keys" Nov 01 23:09:47.645592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.645523 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" containerName="revoke-keys" Nov 01 23:09:47.645592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.645562 4981 cpu_manager.go:414] "containerMap: removing container" podUID="80621398-2493-40fc-a370-001e18141226" containerName="revoke-keys" Nov 01 23:09:47.645592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.645567 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80621398-2493-40fc-a370-001e18141226" containerName="revoke-keys" Nov 01 23:09:47.645710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.645691 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="80621398-2493-40fc-a370-001e18141226" containerName="revoke-keys" Nov 01 23:09:47.645710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.645706 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" containerName="revoke-keys" Nov 01 23:09:47.649779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.649759 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.667159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.667135 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:09:47.668319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.668299 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:09:47.673675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.673652 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v"] Nov 01 23:09:47.724835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.724803 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0a77d-578f-4665-836a-7c0499481396-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.725025 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.724876 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0a77d-578f-4665-836a-7c0499481396-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.725092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.725019 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nd2d\" (UniqueName: \"kubernetes.io/projected/adb0a77d-578f-4665-836a-7c0499481396-kube-api-access-5nd2d\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.826288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.826245 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0a77d-578f-4665-836a-7c0499481396-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.826478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.826332 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0a77d-578f-4665-836a-7c0499481396-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.826478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.826406 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nd2d\" (UniqueName: \"kubernetes.io/projected/adb0a77d-578f-4665-836a-7c0499481396-kube-api-access-5nd2d\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.827071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.827042 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0a77d-578f-4665-836a-7c0499481396-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.828759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.828738 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0a77d-578f-4665-836a-7c0499481396-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.842624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.842600 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nd2d\" (UniqueName: \"kubernetes.io/projected/adb0a77d-578f-4665-836a-7c0499481396-kube-api-access-5nd2d\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.965081 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.965044 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:47.997520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:47.997489 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:48.258970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:48.258902 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v"] Nov 01 23:09:48.373226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:48.373190 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" event={"ID":"adb0a77d-578f-4665-836a-7c0499481396","Type":"ContainerStarted","Data":"9faa8c282aeef259e27a6ad585042c8b0e7beabe33a45b144569c280dae3901b"} Nov 01 23:09:48.976506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:48.976475 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:49.383946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:49.383858 4981 generic.go:356] "Generic (PLEG): container finished" podID="adb0a77d-578f-4665-836a-7c0499481396" containerID="51efe787ebf57f69afc60dd8b0a5d84fe86a0588ac00aa32bd189a11486ae1e2" exitCode=0 Nov 01 23:09:49.384094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:49.383948 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" event={"ID":"adb0a77d-578f-4665-836a-7c0499481396","Type":"ContainerDied","Data":"51efe787ebf57f69afc60dd8b0a5d84fe86a0588ac00aa32bd189a11486ae1e2"} Nov 01 23:09:49.790145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:49.790111 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:50.099538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.099445 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:50.204362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.204331 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:50.321717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.321681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:50.621597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.614114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:50.765996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.765967 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:50.836044 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.835998 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adb0a77d-578f-4665-836a-7c0499481396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0a77d-578f-4665-836a-7c0499481396-maas-api-xks-ca\") pod \"adb0a77d-578f-4665-836a-7c0499481396\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " Nov 01 23:09:50.836270 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.836092 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb0a77d-578f-4665-836a-7c0499481396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0a77d-578f-4665-836a-7c0499481396-maas-api-openshift-ca\") pod \"adb0a77d-578f-4665-836a-7c0499481396\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " Nov 01 23:09:50.836510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.836487 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb0a77d-578f-4665-836a-7c0499481396-kube-api-access-5nd2d\" (UniqueName: \"kubernetes.io/projected/adb0a77d-578f-4665-836a-7c0499481396-kube-api-access-5nd2d\") pod \"adb0a77d-578f-4665-836a-7c0499481396\" (UID: \"adb0a77d-578f-4665-836a-7c0499481396\") " Nov 01 23:09:50.836612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.836507 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb0a77d-578f-4665-836a-7c0499481396-maas-api-openshift-ca" pod "adb0a77d-578f-4665-836a-7c0499481396" (UID: "adb0a77d-578f-4665-836a-7c0499481396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:50.837118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.837090 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adb0a77d-578f-4665-836a-7c0499481396-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:50.838220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.838194 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb0a77d-578f-4665-836a-7c0499481396-maas-api-xks-ca" pod "adb0a77d-578f-4665-836a-7c0499481396" (UID: "adb0a77d-578f-4665-836a-7c0499481396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:09:50.839124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.839106 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb0a77d-578f-4665-836a-7c0499481396-kube-api-access-5nd2d" pod "adb0a77d-578f-4665-836a-7c0499481396" (UID: "adb0a77d-578f-4665-836a-7c0499481396"). InnerVolumeSpecName "kube-api-access-5nd2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:50.939156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.939119 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nd2d\" (UniqueName: \"kubernetes.io/projected/adb0a77d-578f-4665-836a-7c0499481396-kube-api-access-5nd2d\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:50.939156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:50.939152 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adb0a77d-578f-4665-836a-7c0499481396-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:51.404086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:51.404057 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" Nov 01 23:09:51.404491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:51.404087 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v" event={"ID":"adb0a77d-578f-4665-836a-7c0499481396","Type":"ContainerDied","Data":"9faa8c282aeef259e27a6ad585042c8b0e7beabe33a45b144569c280dae3901b"} Nov 01 23:09:51.404491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:51.404147 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9faa8c282aeef259e27a6ad585042c8b0e7beabe33a45b144569c280dae3901b" Nov 01 23:09:51.444608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:51.444553 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" podUID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:09:51.479330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:51.479283 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:52.170539 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.170511 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:52.428282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.428252 4981 generic.go:356] "Generic (PLEG): container finished" podID="00290935-a1cc-4b90-b259-de633a083de1" containerID="e7be2de8a8ec2c64cb8f3dbaad546b12cbeb946a1b2affd1cb538bd9ba3dc4e8" exitCode=0 Nov 01 23:09:52.428683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.428329 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" event={"ID":"00290935-a1cc-4b90-b259-de633a083de1","Type":"ContainerDied","Data":"e7be2de8a8ec2c64cb8f3dbaad546b12cbeb946a1b2affd1cb538bd9ba3dc4e8"} Nov 01 23:09:52.473283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.473257 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 23:09:52.571287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571256 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-certs\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.571447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571298 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-kube-api-access-2hcw5\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-kube-api-access-2hcw5\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.571447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571349 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00290935-a1cc-4b90-b259-de633a083de1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00290935-a1cc-4b90-b259-de633a083de1-istiod-ca-cert\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.571447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571381 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-istio-token\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-istio-token\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.571447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571422 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-socket\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.571646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571519 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00290935-a1cc-4b90-b259-de633a083de1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00290935-a1cc-4b90-b259-de633a083de1-istio-podinfo\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.571646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571568 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-certs" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.571646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571629 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-envoy\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.571782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571721 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-data\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.572233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571854 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-socket" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.572233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571865 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00290935-a1cc-4b90-b259-de633a083de1-istiod-ca-cert" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:52.572233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.571877 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-credential-socket\") pod \"00290935-a1cc-4b90-b259-de633a083de1\" (UID: \"00290935-a1cc-4b90-b259-de633a083de1\") " Nov 01 23:09:52.572233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.572062 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-data" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.572233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.572199 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-credential-socket" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.572671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.572643 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.572819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.572674 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.572819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.572691 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.572819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.572706 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00290935-a1cc-4b90-b259-de633a083de1-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.572819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.572722 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.573718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.573694 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-kube-api-access-2hcw5" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "kube-api-access-2hcw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:52.574016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.573997 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00290935-a1cc-4b90-b259-de633a083de1-istio-podinfo" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:09:52.574183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.574163 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-envoy" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:09:52.574647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.574629 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-istio-token" pod "00290935-a1cc-4b90-b259-de633a083de1" (UID: "00290935-a1cc-4b90-b259-de633a083de1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:52.674353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.674315 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00290935-a1cc-4b90-b259-de633a083de1-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.674353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.674348 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hcw5\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-kube-api-access-2hcw5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.674540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.674362 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00290935-a1cc-4b90-b259-de633a083de1-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.674540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.674374 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00290935-a1cc-4b90-b259-de633a083de1-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:52.821834 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:52.821782 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:53.025066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.024989 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc"] Nov 01 23:09:53.029124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.029102 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0d3bbd-vpkcc"] Nov 01 23:09:53.438822 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.438788 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" event={"ID":"00290935-a1cc-4b90-b259-de633a083de1","Type":"ContainerDied","Data":"219e553f127f90df07660072d89b5124c686c0ef288c27306dfb80243987f909"} Nov 01 23:09:53.438822 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.438809 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp" Nov 01 23:09:53.439317 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.438852 4981 scope.go:122] "RemoveContainer" containerID="e7be2de8a8ec2c64cb8f3dbaad546b12cbeb946a1b2affd1cb538bd9ba3dc4e8" Nov 01 23:09:53.463251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.463227 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp"] Nov 01 23:09:53.467374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.467352 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-788d7b-openshift-default-7c9484856c-l5tqp"] Nov 01 23:09:53.654656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:53.654623 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:54.074029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:54.074000 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:54.682006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:54.681969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:54.738493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:54.738461 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00290935-a1cc-4b90-b259-de633a083de1" path="/var/lib/kubelet/pods/00290935-a1cc-4b90-b259-de633a083de1/volumes" Nov 01 23:09:54.738957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:54.738941 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a41e36f-ca5e-4e8a-b322-c801592f61a4" path="/var/lib/kubelet/pods/2a41e36f-ca5e-4e8a-b322-c801592f61a4/volumes" Nov 01 23:09:55.281165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:55.281129 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:55.516902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:55.516865 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:55.619334 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:55.619248 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:55.647191 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:55.647165 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8"] Nov 01 23:09:55.651682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:55.651656 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-927b68-cxsv8"] Nov 01 23:09:55.709477 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:55.709442 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:55.820261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:55.820228 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:56.045871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.045827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:56.485643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.485610 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:56.739629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.739554 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba35e673-963e-4221-8503-2eb58e6ca473" path="/var/lib/kubelet/pods/ba35e673-963e-4221-8503-2eb58e6ca473/volumes" Nov 01 23:09:56.890089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.890064 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 23:09:56.905132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.905103 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-plugins-config-volume\") pod \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " Nov 01 23:09:56.905292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.905166 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-kube-api-access-bcg5q\" (UniqueName: \"kubernetes.io/projected/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-kube-api-access-bcg5q\") pod \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\" (UID: \"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd\") " Nov 01 23:09:56.905512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.905488 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-plugins-config-volume" pod "4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" (UID: "4e5f7521-0a94-4d62-b9d5-772c9d1f62dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:09:56.905685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.905666 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:56.907411 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:56.907382 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-kube-api-access-bcg5q" pod "4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" (UID: "4e5f7521-0a94-4d62-b9d5-772c9d1f62dd"). InnerVolumeSpecName "kube-api-access-bcg5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:09:57.006982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.006898 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcg5q\" (UniqueName: \"kubernetes.io/projected/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd-kube-api-access-bcg5q\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:09:57.359708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.359627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:57.588000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.587964 4981 generic.go:356] "Generic (PLEG): container finished" podID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" containerID="16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b" exitCode=0 Nov 01 23:09:57.588203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.588008 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" Nov 01 23:09:57.588203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.588034 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" event={"ID":"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd","Type":"ContainerDied","Data":"16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b"} Nov 01 23:09:57.588203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.588099 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6" event={"ID":"4e5f7521-0a94-4d62-b9d5-772c9d1f62dd","Type":"ContainerDied","Data":"9620bb11e20df85072d584e10874ac127d70538a93899b373fda03134b9296e4"} Nov 01 23:09:57.588203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.588131 4981 scope.go:122] "RemoveContainer" containerID="16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b" Nov 01 23:09:57.600262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.600243 4981 scope.go:122] "RemoveContainer" containerID="16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b" Nov 01 23:09:57.600555 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:57.600535 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b\": container with ID starting with 16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b not found: ID does not exist" containerID="16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b" Nov 01 23:09:57.600655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.600563 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b"} err="failed to get container status \"16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b\": rpc error: code = NotFound desc = could not find container \"16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b\": container with ID starting with 16f6bb7dea0cbba13478e1cc6f25c07e30a094031d1518e662da7bc590ad952b not found: ID does not exist" Nov 01 23:09:57.617939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.617881 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6"] Nov 01 23:09:57.624589 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.624554 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-788d7b-7f6b94b8f7-7pmk6"] Nov 01 23:09:57.662370 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:09:57.662337 4981 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5f7521_0a94_4d62_b9d5_772c9d1f62dd.slice/crio-9620bb11e20df85072d584e10874ac127d70538a93899b373fda03134b9296e4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5f7521_0a94_4d62_b9d5_772c9d1f62dd.slice\": RecentStats: unable to find data in memory cache]" Nov 01 23:09:57.764019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.763986 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf"] Nov 01 23:09:57.815425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.815392 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:57.919812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:57.919730 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:58.375689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:58.375653 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:58.737995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:58.737961 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" path="/var/lib/kubelet/pods/4e5f7521-0a94-4d62-b9d5-772c9d1f62dd/volumes" Nov 01 23:09:59.156816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:59.156727 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:09:59.871527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:09:59.871491 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:00.553251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.553195 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:00.643320 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.643284 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts"] Nov 01 23:10:00.644306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644282 4981 cpu_manager.go:414] "containerMap: removing container" podUID="adb0a77d-578f-4665-836a-7c0499481396" containerName="revoke-keys" Nov 01 23:10:00.644306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644303 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb0a77d-578f-4665-836a-7c0499481396" containerName="revoke-keys" Nov 01 23:10:00.644497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644349 4981 cpu_manager.go:414] "containerMap: removing container" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" Nov 01 23:10:00.644497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644358 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" Nov 01 23:10:00.644497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644378 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" containerName="payload-processing" Nov 01 23:10:00.644497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644389 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" containerName="payload-processing" Nov 01 23:10:00.644827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644544 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="00290935-a1cc-4b90-b259-de633a083de1" containerName="istio-proxy" Nov 01 23:10:00.644827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644555 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5f7521-0a94-4d62-b9d5-772c9d1f62dd" containerName="payload-processing" Nov 01 23:10:00.644827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.644568 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb0a77d-578f-4665-836a-7c0499481396" containerName="revoke-keys" Nov 01 23:10:00.650123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.650103 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:00.654824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.654800 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-46ce02\"" Nov 01 23:10:00.658674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.658645 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts"] Nov 01 23:10:00.666820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.666796 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-46ce02-dockercfg-9czp8\"" Nov 01 23:10:00.669527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.669505 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sdd6\" (UniqueName: \"kubernetes.io/projected/c7697c68-e0f4-4913-9d4c-1294ad63a74a-kube-api-access-9sdd6\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:00.669773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.669752 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7697c68-e0f4-4913-9d4c-1294ad63a74a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:00.683025 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.682997 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:00.785725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.785670 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7697c68-e0f4-4913-9d4c-1294ad63a74a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:00.801608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.797037 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sdd6\" (UniqueName: \"kubernetes.io/projected/c7697c68-e0f4-4913-9d4c-1294ad63a74a-kube-api-access-9sdd6\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:00.801608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.791479 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7697c68-e0f4-4913-9d4c-1294ad63a74a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:00.851730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.844746 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sdd6\" (UniqueName: \"kubernetes.io/projected/c7697c68-e0f4-4913-9d4c-1294ad63a74a-kube-api-access-9sdd6\") pod \"payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:00.931034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.930991 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw"] Nov 01 23:10:00.931287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.931255 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" podUID="fba856d5-aa40-41f9-bdbd-ab43935f23fa" containerName="maas-api" containerID="cri-o://e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430" gracePeriod=30 Nov 01 23:10:00.962212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.962169 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:00.970889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:00.970861 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:01.196601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.193792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:01.348637 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:01.348567 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7697c68_e0f4_4913_9d4c_1294ad63a74a.slice/crio-186c665a2d709a84eb8e9a616a978147af78ef2cb26675bd3ff073fc7360b71e WatchSource:0}: Error finding container 186c665a2d709a84eb8e9a616a978147af78ef2cb26675bd3ff073fc7360b71e: Status 404 returned error can't find the container with id 186c665a2d709a84eb8e9a616a978147af78ef2cb26675bd3ff073fc7360b71e Nov 01 23:10:01.353857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.352983 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:01.363124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.362969 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts"] Nov 01 23:10:01.475794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.475768 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 23:10:01.481105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.481076 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v"] Nov 01 23:10:01.512180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.512140 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d73b25-dkx9v"] Nov 01 23:10:01.534509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.534429 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba856d5-aa40-41f9-bdbd-ab43935f23fa-kube-api-access-nwkhb\" (UniqueName: \"kubernetes.io/projected/fba856d5-aa40-41f9-bdbd-ab43935f23fa-kube-api-access-nwkhb\") pod \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " Nov 01 23:10:01.534741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.534522 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-maas-api-tls\") pod \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " Nov 01 23:10:01.534741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.534637 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-metrics-certs\") pod \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " Nov 01 23:10:01.534838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.534813 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba856d5-aa40-41f9-bdbd-ab43935f23fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba856d5-aa40-41f9-bdbd-ab43935f23fa-openshift-service-ca\") pod \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\" (UID: \"fba856d5-aa40-41f9-bdbd-ab43935f23fa\") " Nov 01 23:10:01.535510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.535319 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba856d5-aa40-41f9-bdbd-ab43935f23fa-openshift-service-ca" pod "fba856d5-aa40-41f9-bdbd-ab43935f23fa" (UID: "fba856d5-aa40-41f9-bdbd-ab43935f23fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:01.535510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.535478 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fba856d5-aa40-41f9-bdbd-ab43935f23fa-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:01.537090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.537064 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba856d5-aa40-41f9-bdbd-ab43935f23fa-kube-api-access-nwkhb" pod "fba856d5-aa40-41f9-bdbd-ab43935f23fa" (UID: "fba856d5-aa40-41f9-bdbd-ab43935f23fa"). InnerVolumeSpecName "kube-api-access-nwkhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:01.537352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.537325 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-metrics-certs" pod "fba856d5-aa40-41f9-bdbd-ab43935f23fa" (UID: "fba856d5-aa40-41f9-bdbd-ab43935f23fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:01.537424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.537339 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-maas-api-tls" pod "fba856d5-aa40-41f9-bdbd-ab43935f23fa" (UID: "fba856d5-aa40-41f9-bdbd-ab43935f23fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:01.624032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.623994 4981 generic.go:356] "Generic (PLEG): container finished" podID="fba856d5-aa40-41f9-bdbd-ab43935f23fa" containerID="e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430" exitCode=0 Nov 01 23:10:01.624536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.624044 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" Nov 01 23:10:01.624536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.624073 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" event={"ID":"fba856d5-aa40-41f9-bdbd-ab43935f23fa","Type":"ContainerDied","Data":"e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430"} Nov 01 23:10:01.624536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.624135 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw" event={"ID":"fba856d5-aa40-41f9-bdbd-ab43935f23fa","Type":"ContainerDied","Data":"659107c846aecbd2f55ebea629aaebfa376a23858cd8e32bb9fb72d69d22c494"} Nov 01 23:10:01.624536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.624167 4981 scope.go:122] "RemoveContainer" containerID="e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430" Nov 01 23:10:01.626268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.626239 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" event={"ID":"c7697c68-e0f4-4913-9d4c-1294ad63a74a","Type":"ContainerStarted","Data":"a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780"} Nov 01 23:10:01.626390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.626298 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" event={"ID":"c7697c68-e0f4-4913-9d4c-1294ad63a74a","Type":"ContainerStarted","Data":"186c665a2d709a84eb8e9a616a978147af78ef2cb26675bd3ff073fc7360b71e"} Nov 01 23:10:01.626390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.626355 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:01.636519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.636495 4981 scope.go:122] "RemoveContainer" containerID="e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430" Nov 01 23:10:01.636731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.636687 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:01.636731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.636714 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkhb\" (UniqueName: \"kubernetes.io/projected/fba856d5-aa40-41f9-bdbd-ab43935f23fa-kube-api-access-nwkhb\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:01.636731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.636725 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fba856d5-aa40-41f9-bdbd-ab43935f23fa-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:01.636943 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:01.636920 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430\": container with ID starting with e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430 not found: ID does not exist" containerID="e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430" Nov 01 23:10:01.636996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.636952 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430"} err="failed to get container status \"e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430\": rpc error: code = NotFound desc = could not find container \"e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430\": container with ID starting with e53e870ad7fa01d31b742408694fe94d414517b95fd909b3eba4b4c28fc3f430 not found: ID does not exist" Nov 01 23:10:01.665629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.665482 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" podStartSLOduration=1.665464826 podStartE2EDuration="1.665464826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:01.6615189 +0000 UTC m=+2767.089396830" watchObservedRunningTime="2026-11-01 23:10:01.665464826 +0000 UTC m=+2767.093342756" Nov 01 23:10:01.685315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.685271 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:01.695568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.695530 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw"] Nov 01 23:10:01.702610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.702562 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d73b25-7ffdfbbf74-q66sw"] Nov 01 23:10:01.909964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:01.909928 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:02.620593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:02.620551 4981 scope.go:122] "RemoveContainer" containerID="3a4ccb614fa6f0fda3002e30061a9b2b413922fe9e5652cd3f91902a7a2b5931" Nov 01 23:10:02.632199 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:02.632180 4981 scope.go:122] "RemoveContainer" containerID="a1b67b5859878ee2618efba654fc11d4ad98b57fc0778e95c36cc5989434164f" Nov 01 23:10:02.643371 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:02.643349 4981 scope.go:122] "RemoveContainer" containerID="73e2aa10cabfdf2a7fa2d7baac8571ace3e195271a8852030cf2b1ba4d164589" Nov 01 23:10:02.737185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:02.737154 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb0a77d-578f-4665-836a-7c0499481396" path="/var/lib/kubelet/pods/adb0a77d-578f-4665-836a-7c0499481396/volumes" Nov 01 23:10:02.737823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:02.737808 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba856d5-aa40-41f9-bdbd-ab43935f23fa" path="/var/lib/kubelet/pods/fba856d5-aa40-41f9-bdbd-ab43935f23fa/volumes" Nov 01 23:10:02.764390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:02.764355 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" podUID="6adbe704-d36b-4285-881d-032098ab3cff" containerName="payload-pre-processing" containerID="cri-o://4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea" gracePeriod=25 Nov 01 23:10:02.790138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:02.790108 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:03.173752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:03.173539 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:03.270847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:03.270819 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:03.377887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:03.377853 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:03.562450 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:03.562360 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:03.940943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:03.940876 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:04.049109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.049075 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9"] Nov 01 23:10:04.052946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.052922 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf23f11c-mnmt9"] Nov 01 23:10:04.272293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.272207 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:04.383953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.383922 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:04.618475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.618382 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk"] Nov 01 23:10:04.618715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.618691 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" containerID="cri-o://560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334" gracePeriod=30 Nov 01 23:10:04.625000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.624975 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq"] Nov 01 23:10:04.625199 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.625177 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" containerID="cri-o://a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed" gracePeriod=30 Nov 01 23:10:04.628442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.628414 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:04.816600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.798708 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867cd23e-8b02-4690-8448-aa5eec26500a" path="/var/lib/kubelet/pods/867cd23e-8b02-4690-8448-aa5eec26500a/volumes" Nov 01 23:10:04.893650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:04.893032 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:05.792473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:05.792440 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:06.587840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:06.587803 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:07.373071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:07.373039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:08.165870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.165830 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:08.321906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.321871 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5"] Nov 01 23:10:08.322982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.322960 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fba856d5-aa40-41f9-bdbd-ab43935f23fa" containerName="maas-api" Nov 01 23:10:08.323095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.322983 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba856d5-aa40-41f9-bdbd-ab43935f23fa" containerName="maas-api" Nov 01 23:10:08.323210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.323198 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba856d5-aa40-41f9-bdbd-ab43935f23fa" containerName="maas-api" Nov 01 23:10:08.331620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.331593 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.334512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.334478 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5"] Nov 01 23:10:08.336463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.336434 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:10:08.336604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.336525 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:10:08.405909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.405872 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.406314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.405993 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.406314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.406196 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb75l\" (UniqueName: \"kubernetes.io/projected/0257d23e-b180-43b1-bfed-5a499fc4fb63-kube-api-access-qb75l\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.477064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.477024 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:08.507292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.507253 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb75l\" (UniqueName: \"kubernetes.io/projected/0257d23e-b180-43b1-bfed-5a499fc4fb63-kube-api-access-qb75l\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.507455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.507363 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.507455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.507409 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.508107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.508085 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.510074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.510049 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.517419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.517387 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb75l\" (UniqueName: \"kubernetes.io/projected/0257d23e-b180-43b1-bfed-5a499fc4fb63-kube-api-access-qb75l\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.578332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.578297 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:08.652235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.652199 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:08.770642 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.770539 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd"] Nov 01 23:10:08.771099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.770806 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" containerID="cri-o://715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11" gracePeriod=30 Nov 01 23:10:08.803587 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:08.803539 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:09.014521 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:09.014477 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0257d23e_b180_43b1_bfed_5a499fc4fb63.slice/crio-faccdd29e73612a70d93ecfc25e62885a21112d093e149b267851d824fd9a8f2 WatchSource:0}: Error finding container faccdd29e73612a70d93ecfc25e62885a21112d093e149b267851d824fd9a8f2: Status 404 returned error can't find the container with id faccdd29e73612a70d93ecfc25e62885a21112d093e149b267851d824fd9a8f2 Nov 01 23:10:09.016410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.016388 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5"] Nov 01 23:10:09.270826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.270792 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:09.387256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.387226 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:09.509885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.509849 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc"] Nov 01 23:10:09.514549 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.514528 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.522812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.522784 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc"] Nov 01 23:10:09.638007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.637980 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9hr\" (UniqueName: \"kubernetes.io/projected/aa17d216-8f94-4423-817c-fa6ea23923ce-kube-api-access-nd9hr\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.638129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.638100 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.638221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.638205 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.709236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.709198 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" event={"ID":"0257d23e-b180-43b1-bfed-5a499fc4fb63","Type":"ContainerStarted","Data":"a53cb9bbe1ddfc847006250e580b0998817ad49816a216f57aa22fcd7ce199f2"} Nov 01 23:10:09.709398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.709252 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" event={"ID":"0257d23e-b180-43b1-bfed-5a499fc4fb63","Type":"ContainerStarted","Data":"faccdd29e73612a70d93ecfc25e62885a21112d093e149b267851d824fd9a8f2"} Nov 01 23:10:09.735764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.735712 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" podStartSLOduration=1.7356953160000002 podStartE2EDuration="1.735695316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:09.731827369 +0000 UTC m=+2775.159705302" watchObservedRunningTime="2026-11-01 23:10:09.735695316 +0000 UTC m=+2775.163573248" Nov 01 23:10:09.740379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.740332 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9hr\" (UniqueName: \"kubernetes.io/projected/aa17d216-8f94-4423-817c-fa6ea23923ce-kube-api-access-nd9hr\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.740766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.740741 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.740877 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.740863 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.741493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.741471 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.743047 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.743029 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.750899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.750872 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9hr\" (UniqueName: \"kubernetes.io/projected/aa17d216-8f94-4423-817c-fa6ea23923ce-kube-api-access-nd9hr\") pod \"maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.842566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.842531 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:09.892318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.891866 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 23:10:09.944228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944199 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-kube-api-access-8w7vg\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-kube-api-access-8w7vg\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.944448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944271 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-envoy\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.944448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944326 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-socket\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.944448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944391 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-credential-socket\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.944448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944422 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-istio-token\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-istio-token\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.944716 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944487 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5987c39-5892-4330-abf3-1694e5668153-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5987c39-5892-4330-abf3-1694e5668153-istiod-ca-cert\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.944778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944747 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5987c39-5892-4330-abf3-1694e5668153-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5987c39-5892-4330-abf3-1694e5668153-istio-podinfo\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.944843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944827 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-data\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.945314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.944889 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-certs\") pod \"c5987c39-5892-4330-abf3-1694e5668153\" (UID: \"c5987c39-5892-4330-abf3-1694e5668153\") " Nov 01 23:10:09.945948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.945924 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-certs" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.946953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.946655 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-socket" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.947335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.947286 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5987c39-5892-4330-abf3-1694e5668153-istiod-ca-cert" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:09.947513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.947482 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-data" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.947685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.947657 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-credential-socket" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.948661 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.948565 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-kube-api-access-8w7vg" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "kube-api-access-8w7vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:09.950000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.949967 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-istio-token" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:09.950463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.950433 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-envoy" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:09.953376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.953351 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5987c39-5892-4330-abf3-1694e5668153-istio-podinfo" pod "c5987c39-5892-4330-abf3-1694e5668153" (UID: "c5987c39-5892-4330-abf3-1694e5668153"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:09.972984 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:09.972958 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 23:10:10.046906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.046868 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefeccec-8cb6-433e-99dc-195307e4621a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefeccec-8cb6-433e-99dc-195307e4621a-istiod-ca-cert\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.046932 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-istio-token\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-istio-token\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047028 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-envoy\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047098 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-credential-socket\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047144 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-certs\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047202 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-data\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047298 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefeccec-8cb6-433e-99dc-195307e4621a-istiod-ca-cert" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:10.047432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047329 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-socket\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047455 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fefeccec-8cb6-433e-99dc-195307e4621a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefeccec-8cb6-433e-99dc-195307e4621a-istio-podinfo\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047488 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-kube-api-access-t2tpg\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-kube-api-access-t2tpg\") pod \"fefeccec-8cb6-433e-99dc-195307e4621a\" (UID: \"fefeccec-8cb6-433e-99dc-195307e4621a\") " Nov 01 23:10:10.047781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.047634 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-credential-socket" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.048083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048056 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-data" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.048218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048165 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-certs" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.048289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048251 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5987c39-5892-4330-abf3-1694e5668153-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048266 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefeccec-8cb6-433e-99dc-195307e4621a-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048265 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-socket" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048277 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048317 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048334 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5987c39-5892-4330-abf3-1694e5668153-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048349 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048365 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048379 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w7vg\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-kube-api-access-8w7vg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048394 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048410 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048426 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5987c39-5892-4330-abf3-1694e5668153-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.048506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.048440 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5987c39-5892-4330-abf3-1694e5668153-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.050735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.050710 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-envoy" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:10.050852 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.050743 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fefeccec-8cb6-433e-99dc-195307e4621a-istio-podinfo" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:10.051110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.051085 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-istio-token" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:10.051285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.051266 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-kube-api-access-t2tpg" pod "fefeccec-8cb6-433e-99dc-195307e4621a" (UID: "fefeccec-8cb6-433e-99dc-195307e4621a"). InnerVolumeSpecName "kube-api-access-t2tpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:10.149540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.149510 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefeccec-8cb6-433e-99dc-195307e4621a-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.149707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.149542 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tpg\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-kube-api-access-t2tpg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.149707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.149557 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefeccec-8cb6-433e-99dc-195307e4621a-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.149707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.149570 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.149707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.149621 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.149707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.149635 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefeccec-8cb6-433e-99dc-195307e4621a-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:10.149707 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:10.149636 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa17d216_8f94_4423_817c_fa6ea23923ce.slice/crio-e48120deb1e8d09c33d891444933678c306394f4a59785b8b11bdab6116fbe4d WatchSource:0}: Error finding container e48120deb1e8d09c33d891444933678c306394f4a59785b8b11bdab6116fbe4d: Status 404 returned error can't find the container with id e48120deb1e8d09c33d891444933678c306394f4a59785b8b11bdab6116fbe4d Nov 01 23:10:10.150278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.150225 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc"] Nov 01 23:10:10.430793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.430762 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:10.663434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.663400 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:10.719458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.719355 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" event={"ID":"aa17d216-8f94-4423-817c-fa6ea23923ce","Type":"ContainerStarted","Data":"be4b7645b52fa6c7265add943e594fb5af846b022814edaf687b961de69488c4"} Nov 01 23:10:10.719458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.719416 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" event={"ID":"aa17d216-8f94-4423-817c-fa6ea23923ce","Type":"ContainerStarted","Data":"e48120deb1e8d09c33d891444933678c306394f4a59785b8b11bdab6116fbe4d"} Nov 01 23:10:10.720986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.720958 4981 generic.go:356] "Generic (PLEG): container finished" podID="c5987c39-5892-4330-abf3-1694e5668153" containerID="560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334" exitCode=0 Nov 01 23:10:10.721117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.721003 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" Nov 01 23:10:10.721117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.721005 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" event={"ID":"c5987c39-5892-4330-abf3-1694e5668153","Type":"ContainerDied","Data":"560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334"} Nov 01 23:10:10.721117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.721079 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk" event={"ID":"c5987c39-5892-4330-abf3-1694e5668153","Type":"ContainerDied","Data":"be01546f73d9ef5f93b12dd1a4bd8575f41266483222c6f1479e1963a1400feb"} Nov 01 23:10:10.721296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.721130 4981 scope.go:122] "RemoveContainer" containerID="560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334" Nov 01 23:10:10.722863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.722840 4981 generic.go:356] "Generic (PLEG): container finished" podID="fefeccec-8cb6-433e-99dc-195307e4621a" containerID="a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed" exitCode=0 Nov 01 23:10:10.722973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.722881 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" Nov 01 23:10:10.722973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.722950 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" event={"ID":"fefeccec-8cb6-433e-99dc-195307e4621a","Type":"ContainerDied","Data":"a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed"} Nov 01 23:10:10.723091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.722988 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq" event={"ID":"fefeccec-8cb6-433e-99dc-195307e4621a","Type":"ContainerDied","Data":"0557fb24b7a8153be5f506f00f0fd849ce5b270ee40fbc16e2b9cfe378b381d0"} Nov 01 23:10:10.726133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.726107 4981 generic.go:356] "Generic (PLEG): container finished" podID="0257d23e-b180-43b1-bfed-5a499fc4fb63" containerID="a53cb9bbe1ddfc847006250e580b0998817ad49816a216f57aa22fcd7ce199f2" exitCode=0 Nov 01 23:10:10.726245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.726183 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" event={"ID":"0257d23e-b180-43b1-bfed-5a499fc4fb63","Type":"ContainerDied","Data":"a53cb9bbe1ddfc847006250e580b0998817ad49816a216f57aa22fcd7ce199f2"} Nov 01 23:10:10.736777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.736375 4981 scope.go:122] "RemoveContainer" containerID="560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334" Nov 01 23:10:10.747461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.747240 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" podStartSLOduration=1.7472207260000001 podStartE2EDuration="1.747220726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:10.740724267 +0000 UTC m=+2776.168602200" watchObservedRunningTime="2026-11-01 23:10:10.747220726 +0000 UTC m=+2776.175098659" Nov 01 23:10:10.747461 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:10.747352 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334\": container with ID starting with 560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334 not found: ID does not exist" containerID="560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334" Nov 01 23:10:10.747461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.747422 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334"} err="failed to get container status \"560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334\": rpc error: code = NotFound desc = could not find container \"560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334\": container with ID starting with 560cf94b9ace76142f474a71fe0c3f9c8450a5cc3c3179d48d68e0a79b4ed334 not found: ID does not exist" Nov 01 23:10:10.747461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.747463 4981 scope.go:122] "RemoveContainer" containerID="a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed" Nov 01 23:10:10.767638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.767616 4981 scope.go:122] "RemoveContainer" containerID="a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed" Nov 01 23:10:10.767927 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:10.767904 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed\": container with ID starting with a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed not found: ID does not exist" containerID="a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed" Nov 01 23:10:10.768004 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.767938 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed"} err="failed to get container status \"a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed\": rpc error: code = NotFound desc = could not find container \"a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed\": container with ID starting with a50670d42349f3d74e27681f7558c74d314107ece6754c5078fbe7a3a26a82ed not found: ID does not exist" Nov 01 23:10:10.784263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.784231 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk"] Nov 01 23:10:10.790882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.790858 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-v9qqk"] Nov 01 23:10:10.802451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.802423 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq"] Nov 01 23:10:10.811219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:10.811193 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-927b68-openshift-default-6c675dbcdf-2qmnq"] Nov 01 23:10:11.170128 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:11.170033 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:11.741805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:11.741775 4981 generic.go:356] "Generic (PLEG): container finished" podID="aa17d216-8f94-4423-817c-fa6ea23923ce" containerID="be4b7645b52fa6c7265add943e594fb5af846b022814edaf687b961de69488c4" exitCode=0 Nov 01 23:10:11.744251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:11.741865 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" event={"ID":"aa17d216-8f94-4423-817c-fa6ea23923ce","Type":"ContainerDied","Data":"be4b7645b52fa6c7265add943e594fb5af846b022814edaf687b961de69488c4"} Nov 01 23:10:11.889397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:11.889364 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:11.933109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:11.933085 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:12.042512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.042433 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" podUID="6adbe704-d36b-4285-881d-032098ab3cff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:12.077901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.077868 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-xks-ca\") pod \"0257d23e-b180-43b1-bfed-5a499fc4fb63\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " Nov 01 23:10:12.078095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.077991 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-openshift-ca\") pod \"0257d23e-b180-43b1-bfed-5a499fc4fb63\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " Nov 01 23:10:12.078095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.078044 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0257d23e-b180-43b1-bfed-5a499fc4fb63-kube-api-access-qb75l\" (UniqueName: \"kubernetes.io/projected/0257d23e-b180-43b1-bfed-5a499fc4fb63-kube-api-access-qb75l\") pod \"0257d23e-b180-43b1-bfed-5a499fc4fb63\" (UID: \"0257d23e-b180-43b1-bfed-5a499fc4fb63\") " Nov 01 23:10:12.078367 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.078340 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-openshift-ca" pod "0257d23e-b180-43b1-bfed-5a499fc4fb63" (UID: "0257d23e-b180-43b1-bfed-5a499fc4fb63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:12.078535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.078519 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:12.080022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.079999 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-xks-ca" pod "0257d23e-b180-43b1-bfed-5a499fc4fb63" (UID: "0257d23e-b180-43b1-bfed-5a499fc4fb63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:12.080126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.080108 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0257d23e-b180-43b1-bfed-5a499fc4fb63-kube-api-access-qb75l" pod "0257d23e-b180-43b1-bfed-5a499fc4fb63" (UID: "0257d23e-b180-43b1-bfed-5a499fc4fb63"). InnerVolumeSpecName "kube-api-access-qb75l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:12.180097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.180052 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d23e-b180-43b1-bfed-5a499fc4fb63-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:12.180097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.180094 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb75l\" (UniqueName: \"kubernetes.io/projected/0257d23e-b180-43b1-bfed-5a499fc4fb63-kube-api-access-qb75l\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:12.629510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.629474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:12.639178 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.639150 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:10:12.739259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.739222 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5987c39-5892-4330-abf3-1694e5668153" path="/var/lib/kubelet/pods/c5987c39-5892-4330-abf3-1694e5668153/volumes" Nov 01 23:10:12.739812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.739793 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" path="/var/lib/kubelet/pods/fefeccec-8cb6-433e-99dc-195307e4621a/volumes" Nov 01 23:10:12.753502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.753482 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" Nov 01 23:10:12.753952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.753520 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5" event={"ID":"0257d23e-b180-43b1-bfed-5a499fc4fb63","Type":"ContainerDied","Data":"faccdd29e73612a70d93ecfc25e62885a21112d093e149b267851d824fd9a8f2"} Nov 01 23:10:12.753952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.753569 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faccdd29e73612a70d93ecfc25e62885a21112d093e149b267851d824fd9a8f2" Nov 01 23:10:12.920825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.920798 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:12.995671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.995635 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-xks-ca\") pod \"aa17d216-8f94-4423-817c-fa6ea23923ce\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " Nov 01 23:10:12.995871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.995773 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa17d216-8f94-4423-817c-fa6ea23923ce-kube-api-access-nd9hr\" (UniqueName: \"kubernetes.io/projected/aa17d216-8f94-4423-817c-fa6ea23923ce-kube-api-access-nd9hr\") pod \"aa17d216-8f94-4423-817c-fa6ea23923ce\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " Nov 01 23:10:12.995871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.995831 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-openshift-ca\") pod \"aa17d216-8f94-4423-817c-fa6ea23923ce\" (UID: \"aa17d216-8f94-4423-817c-fa6ea23923ce\") " Nov 01 23:10:12.996203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.996176 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-openshift-ca" pod "aa17d216-8f94-4423-817c-fa6ea23923ce" (UID: "aa17d216-8f94-4423-817c-fa6ea23923ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:12.996556 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.996521 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:12.998073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.998049 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa17d216-8f94-4423-817c-fa6ea23923ce-kube-api-access-nd9hr" pod "aa17d216-8f94-4423-817c-fa6ea23923ce" (UID: "aa17d216-8f94-4423-817c-fa6ea23923ce"). InnerVolumeSpecName "kube-api-access-nd9hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:12.998188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:12.998140 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-xks-ca" pod "aa17d216-8f94-4423-817c-fa6ea23923ce" (UID: "aa17d216-8f94-4423-817c-fa6ea23923ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:13.098059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:13.098022 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa17d216-8f94-4423-817c-fa6ea23923ce-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:13.098059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:13.098057 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd9hr\" (UniqueName: \"kubernetes.io/projected/aa17d216-8f94-4423-817c-fa6ea23923ce-kube-api-access-nd9hr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:13.463375 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:13.463339 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:13.764308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:13.764233 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" Nov 01 23:10:13.764308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:13.764246 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc" event={"ID":"aa17d216-8f94-4423-817c-fa6ea23923ce","Type":"ContainerDied","Data":"e48120deb1e8d09c33d891444933678c306394f4a59785b8b11bdab6116fbe4d"} Nov 01 23:10:13.764308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:13.764287 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48120deb1e8d09c33d891444933678c306394f4a59785b8b11bdab6116fbe4d" Nov 01 23:10:14.049681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.049657 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:10:14.111683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.111649 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-kube-api-access-bvz5n\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-kube-api-access-bvz5n\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.111875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.111708 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-socket\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.111875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.111770 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-envoy\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.111875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.111803 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-certs\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.111875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.111863 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-token\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-token\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.112124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.111890 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-data\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.112193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.112134 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istiod-ca-cert\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.112257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.112228 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-certs" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.112326 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.112219 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-credential-socket\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.112389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.112337 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-podinfo\") pod \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\" (UID: \"7d5ac8ef-364e-4fd4-a832-db34cebe5282\") " Nov 01 23:10:14.112637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.112443 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-credential-socket" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.112637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.112626 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istiod-ca-cert" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:14.112800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.112686 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-data" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.113139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.113111 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.113276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.113142 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.113276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.113157 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.113276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.113172 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.113643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.113620 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-socket" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.114669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.114643 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-kube-api-access-bvz5n" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "kube-api-access-bvz5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:14.115467 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.115032 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-envoy" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:14.115467 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.115047 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-podinfo" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:14.115855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.115827 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-token" pod "7d5ac8ef-364e-4fd4-a832-db34cebe5282" (UID: "7d5ac8ef-364e-4fd4-a832-db34cebe5282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:14.214184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.214148 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.214184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.214174 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvz5n\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-kube-api-access-bvz5n\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.214184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.214185 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.214418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.214194 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.214418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.214203 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d5ac8ef-364e-4fd4-a832-db34cebe5282-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:14.228647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.228614 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72"] Nov 01 23:10:14.354345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.354261 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:14.465603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.464553 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:14.525008 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:14.524974 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:10:14.773324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.773289 4981 generic.go:356] "Generic (PLEG): container finished" podID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerID="715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11" exitCode=0 Nov 01 23:10:14.773688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.773350 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" Nov 01 23:10:14.773688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.773366 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" event={"ID":"7d5ac8ef-364e-4fd4-a832-db34cebe5282","Type":"ContainerDied","Data":"715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11"} Nov 01 23:10:14.773688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.773419 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd" event={"ID":"7d5ac8ef-364e-4fd4-a832-db34cebe5282","Type":"ContainerDied","Data":"f0861e2f8cbd29026954dc43e466fe0faeaeaa791755487f10185a864aa2c123"} Nov 01 23:10:14.773688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.773441 4981 scope.go:122] "RemoveContainer" containerID="715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11" Nov 01 23:10:14.785011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.784980 4981 scope.go:122] "RemoveContainer" containerID="715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11" Nov 01 23:10:14.785306 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:14.785285 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11\": container with ID starting with 715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11 not found: ID does not exist" containerID="715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11" Nov 01 23:10:14.785363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.785315 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11"} err="failed to get container status \"715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11\": rpc error: code = NotFound desc = could not find container \"715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11\": container with ID starting with 715221a7e847f85ab6a3e8a0b860625da6d0b0adccddbd4c39c1442ed3dbed11 not found: ID does not exist" Nov 01 23:10:14.793467 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.793444 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd"] Nov 01 23:10:14.797347 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:14.797324 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d73b25-openshift-default-ddbf676d7-d4kvd"] Nov 01 23:10:15.092017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:15.091939 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:15.507890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:15.507856 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:15.872257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:15.872168 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:16.117330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:16.117282 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:16.736927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:16.736892 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" path="/var/lib/kubelet/pods/7d5ac8ef-364e-4fd4-a832-db34cebe5282/volumes" Nov 01 23:10:16.883172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:16.883131 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:17.761256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:17.761224 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:18.476059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:18.476014 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:19.175418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:19.175364 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:19.592535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:19.592449 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2"] Nov 01 23:10:19.617898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:19.617859 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf"] Nov 01 23:10:19.674207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:19.674176 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:19.774030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:19.773998 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:20.000659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:20.000619 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:20.649791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:20.649754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:21.252823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:21.252788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:21.878803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:21.878769 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:22.042793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.042759 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" podUID="6adbe704-d36b-4285-881d-032098ab3cff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:22.679924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.679896 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8"] Nov 01 23:10:22.680585 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680561 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" Nov 01 23:10:22.680648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680588 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" Nov 01 23:10:22.680648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680612 4981 cpu_manager.go:414] "containerMap: removing container" podUID="aa17d216-8f94-4423-817c-fa6ea23923ce" containerName="revoke-keys" Nov 01 23:10:22.680648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680618 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa17d216-8f94-4423-817c-fa6ea23923ce" containerName="revoke-keys" Nov 01 23:10:22.680648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680628 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" Nov 01 23:10:22.680648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680633 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" Nov 01 23:10:22.680648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680641 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" Nov 01 23:10:22.680648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680645 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" Nov 01 23:10:22.680914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680652 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0257d23e-b180-43b1-bfed-5a499fc4fb63" containerName="revoke-keys" Nov 01 23:10:22.680914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680657 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0257d23e-b180-43b1-bfed-5a499fc4fb63" containerName="revoke-keys" Nov 01 23:10:22.680914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680777 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa17d216-8f94-4423-817c-fa6ea23923ce" containerName="revoke-keys" Nov 01 23:10:22.680914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680786 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5ac8ef-364e-4fd4-a832-db34cebe5282" containerName="istio-proxy" Nov 01 23:10:22.680914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680792 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5987c39-5892-4330-abf3-1694e5668153" containerName="istio-proxy" Nov 01 23:10:22.680914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680799 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0257d23e-b180-43b1-bfed-5a499fc4fb63" containerName="revoke-keys" Nov 01 23:10:22.680914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.680812 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefeccec-8cb6-433e-99dc-195307e4621a" containerName="istio-proxy" Nov 01 23:10:22.684531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.684514 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.688267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.688246 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:10:22.689164 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.689143 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:10:22.692376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.692349 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8"] Nov 01 23:10:22.764459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.764427 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:22.844997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.844286 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.844997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.844351 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.844997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.844483 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnb6\" (UniqueName: \"kubernetes.io/projected/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-kube-api-access-fvnb6\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.947171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.945740 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnb6\" (UniqueName: \"kubernetes.io/projected/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-kube-api-access-fvnb6\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.947171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.945986 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.947171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.946036 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.947171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.946970 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.949765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.949728 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:22.958570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:22.958525 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnb6\" (UniqueName: \"kubernetes.io/projected/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-kube-api-access-fvnb6\") pod \"maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:23.020043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.019813 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:23.078752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.078725 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 23:10:23.250126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.250046 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adbe704-d36b-4285-881d-032098ab3cff-kube-api-access-w4cd6\" (UniqueName: \"kubernetes.io/projected/6adbe704-d36b-4285-881d-032098ab3cff-kube-api-access-w4cd6\") pod \"6adbe704-d36b-4285-881d-032098ab3cff\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " Nov 01 23:10:23.250126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.250119 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adbe704-d36b-4285-881d-032098ab3cff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adbe704-d36b-4285-881d-032098ab3cff-plugins-config-volume\") pod \"6adbe704-d36b-4285-881d-032098ab3cff\" (UID: \"6adbe704-d36b-4285-881d-032098ab3cff\") " Nov 01 23:10:23.250496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.250474 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adbe704-d36b-4285-881d-032098ab3cff-plugins-config-volume" pod "6adbe704-d36b-4285-881d-032098ab3cff" (UID: "6adbe704-d36b-4285-881d-032098ab3cff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:23.250756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.250741 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6adbe704-d36b-4285-881d-032098ab3cff-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:23.252226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.252206 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adbe704-d36b-4285-881d-032098ab3cff-kube-api-access-w4cd6" pod "6adbe704-d36b-4285-881d-032098ab3cff" (UID: "6adbe704-d36b-4285-881d-032098ab3cff"). InnerVolumeSpecName "kube-api-access-w4cd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:23.311094 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:23.311070 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bbb3d1_e4fb_432c_9bd6_58d95b97c7b7.slice/crio-8f6192f469b883b53edd19696d4c934bbccd25c39690168c4ac4a70946838f8c WatchSource:0}: Error finding container 8f6192f469b883b53edd19696d4c934bbccd25c39690168c4ac4a70946838f8c: Status 404 returned error can't find the container with id 8f6192f469b883b53edd19696d4c934bbccd25c39690168c4ac4a70946838f8c Nov 01 23:10:23.313798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.313771 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8"] Nov 01 23:10:23.352199 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.352178 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4cd6\" (UniqueName: \"kubernetes.io/projected/6adbe704-d36b-4285-881d-032098ab3cff-kube-api-access-w4cd6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:23.657742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.657660 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:23.859709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.859674 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" event={"ID":"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7","Type":"ContainerStarted","Data":"97eb21d7537757009ec44276de98c882b38c06f773497a03c2085f40dea952ee"} Nov 01 23:10:23.859894 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.859735 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" event={"ID":"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7","Type":"ContainerStarted","Data":"8f6192f469b883b53edd19696d4c934bbccd25c39690168c4ac4a70946838f8c"} Nov 01 23:10:23.861169 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.861147 4981 generic.go:356] "Generic (PLEG): container finished" podID="6adbe704-d36b-4285-881d-032098ab3cff" containerID="4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea" exitCode=0 Nov 01 23:10:23.861287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.861182 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" Nov 01 23:10:23.861287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.861221 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" event={"ID":"6adbe704-d36b-4285-881d-032098ab3cff","Type":"ContainerDied","Data":"4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea"} Nov 01 23:10:23.861287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.861269 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf" event={"ID":"6adbe704-d36b-4285-881d-032098ab3cff","Type":"ContainerDied","Data":"b24a7243be4dc3b30344666fda5260b5d641f6e9068d118fd919ba123de85745"} Nov 01 23:10:23.861287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.861288 4981 scope.go:122] "RemoveContainer" containerID="4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea" Nov 01 23:10:23.875642 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.874962 4981 scope.go:122] "RemoveContainer" containerID="4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea" Nov 01 23:10:23.876268 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:23.876243 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea\": container with ID starting with 4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea not found: ID does not exist" containerID="4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea" Nov 01 23:10:23.876340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.876275 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea"} err="failed to get container status \"4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea\": rpc error: code = NotFound desc = could not find container \"4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea\": container with ID starting with 4bd60be8ea427e9b43cd0d9aa44596f5ddbbbaf0be38e076af76ada4907da7ea not found: ID does not exist" Nov 01 23:10:23.880855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.880816 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" podStartSLOduration=1.8808038790000001 podStartE2EDuration="1.880803879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:23.875868467 +0000 UTC m=+2789.303746389" watchObservedRunningTime="2026-11-01 23:10:23.880803879 +0000 UTC m=+2789.308681810" Nov 01 23:10:23.900399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.900373 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf"] Nov 01 23:10:23.904055 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:23.904037 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d73b25-7857844d99-g6kxf"] Nov 01 23:10:24.364920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:24.364842 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:24.592837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:24.592787 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" podUID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" containerName="payload-processing" containerID="cri-o://f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01" gracePeriod=25 Nov 01 23:10:24.618889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:24.618819 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" podUID="babb7476-eab9-41b5-9729-c94dd542ad9d" containerName="payload-pre-processing" containerID="cri-o://d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c" gracePeriod=25 Nov 01 23:10:24.736840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:24.736808 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adbe704-d36b-4285-881d-032098ab3cff" path="/var/lib/kubelet/pods/6adbe704-d36b-4285-881d-032098ab3cff/volumes" Nov 01 23:10:24.870800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:24.870721 4981 generic.go:356] "Generic (PLEG): container finished" podID="43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" containerID="97eb21d7537757009ec44276de98c882b38c06f773497a03c2085f40dea952ee" exitCode=0 Nov 01 23:10:24.870800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:24.870752 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" event={"ID":"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7","Type":"ContainerDied","Data":"97eb21d7537757009ec44276de98c882b38c06f773497a03c2085f40dea952ee"} Nov 01 23:10:25.072192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:25.072164 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:25.772808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:25.772773 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:26.136604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.135478 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:26.217594 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.217277 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-openshift-ca\") pod \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " Nov 01 23:10:26.217594 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.217487 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-kube-api-access-fvnb6\" (UniqueName: \"kubernetes.io/projected/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-kube-api-access-fvnb6\") pod \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " Nov 01 23:10:26.217888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.217624 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-xks-ca\") pod \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\" (UID: \"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7\") " Nov 01 23:10:26.222600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.218781 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-openshift-ca" pod "43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" (UID: "43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:26.249599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.242792 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-kube-api-access-fvnb6" pod "43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" (UID: "43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7"). InnerVolumeSpecName "kube-api-access-fvnb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:26.249599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.245513 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-xks-ca" pod "43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" (UID: "43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:26.331600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.331090 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:26.331600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.331116 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:26.331600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.331127 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnb6\" (UniqueName: \"kubernetes.io/projected/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7-kube-api-access-fvnb6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:26.759600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.759432 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:26.902598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.902083 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" event={"ID":"43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7","Type":"ContainerDied","Data":"8f6192f469b883b53edd19696d4c934bbccd25c39690168c4ac4a70946838f8c"} Nov 01 23:10:26.902598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.902148 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f6192f469b883b53edd19696d4c934bbccd25c39690168c4ac4a70946838f8c" Nov 01 23:10:26.902598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:26.902218 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8" Nov 01 23:10:27.768314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:27.768284 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:28.315376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.315337 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf"] Nov 01 23:10:28.316080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.316063 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6adbe704-d36b-4285-881d-032098ab3cff" containerName="payload-pre-processing" Nov 01 23:10:28.316151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.316081 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adbe704-d36b-4285-881d-032098ab3cff" containerName="payload-pre-processing" Nov 01 23:10:28.316151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.316108 4981 cpu_manager.go:414] "containerMap: removing container" podUID="43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" containerName="revoke-keys" Nov 01 23:10:28.316151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.316118 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" containerName="revoke-keys" Nov 01 23:10:28.316289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.316273 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adbe704-d36b-4285-881d-032098ab3cff" containerName="payload-pre-processing" Nov 01 23:10:28.316338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.316302 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" containerName="revoke-keys" Nov 01 23:10:28.320295 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.320267 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.326212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.326188 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:10:28.327194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.327171 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj"] Nov 01 23:10:28.327755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.327735 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:10:28.332300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.332271 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.332782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.332750 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf"] Nov 01 23:10:28.346302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.343129 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj"] Nov 01 23:10:28.373674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.373634 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b665k\" (UniqueName: \"kubernetes.io/projected/da815bca-be05-447f-80a2-90ace52e4e04-kube-api-access-b665k\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.373821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.373693 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnfcg\" (UniqueName: \"kubernetes.io/projected/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-kube-api-access-tnfcg\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.373982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.373957 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.374031 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.374016 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.374141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.374118 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.374202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.374185 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.476064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.475974 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.476064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.476031 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.476366 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.476071 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.476366 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.476118 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.476366 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.476202 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b665k\" (UniqueName: \"kubernetes.io/projected/da815bca-be05-447f-80a2-90ace52e4e04-kube-api-access-b665k\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.476366 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.476235 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnfcg\" (UniqueName: \"kubernetes.io/projected/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-kube-api-access-tnfcg\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.476842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.476815 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.476937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.476920 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.479130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.479106 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.479211 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.479105 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.484196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.484175 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b665k\" (UniqueName: \"kubernetes.io/projected/da815bca-be05-447f-80a2-90ace52e4e04-kube-api-access-b665k\") pod \"maas-api-revoke-sub-376015f8-pxnsf\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.484520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.484501 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnfcg\" (UniqueName: \"kubernetes.io/projected/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-kube-api-access-tnfcg\") pod \"maas-api-revoke-sub-f749f3a1-dq7mj\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.605798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.605751 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:28.643901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.643873 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:28.656651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.655435 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:28.975661 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:28.975633 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda815bca_be05_447f_80a2_90ace52e4e04.slice/crio-1922ad8eb8a5b088093f8911a437098c7e38c6302a86b2ce17a53fd302590531 WatchSource:0}: Error finding container 1922ad8eb8a5b088093f8911a437098c7e38c6302a86b2ce17a53fd302590531: Status 404 returned error can't find the container with id 1922ad8eb8a5b088093f8911a437098c7e38c6302a86b2ce17a53fd302590531 Nov 01 23:10:28.977356 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.977332 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf"] Nov 01 23:10:28.989218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:28.989191 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj"] Nov 01 23:10:28.989513 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:28.989496 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1f7468_8561_40a0_ae39_70aafe1a7cb8.slice/crio-c59c99ca9267269d40e5600ac67d49ed7c949749795440054d5406f7f24454c5 WatchSource:0}: Error finding container c59c99ca9267269d40e5600ac67d49ed7c949749795440054d5406f7f24454c5: Status 404 returned error can't find the container with id c59c99ca9267269d40e5600ac67d49ed7c949749795440054d5406f7f24454c5 Nov 01 23:10:29.250676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.250630 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" containerID="cri-o://38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d" gracePeriod=45 Nov 01 23:10:29.265102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.265078 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:29.317643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.317590 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8"] Nov 01 23:10:29.318123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.317816 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" podUID="6c81b530-835e-4ec1-aacb-b05d5b8efc37" containerName="payload-processing" containerID="cri-o://92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5" gracePeriod=30 Nov 01 23:10:29.326019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.325932 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw"] Nov 01 23:10:29.326156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.326131 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" podUID="9edf3a9a-a7ca-4a51-ba63-eb40609e5357" containerName="payload-pre-processing" containerID="cri-o://bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07" gracePeriod=30 Nov 01 23:10:29.366546 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.366525 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:29.396310 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.396279 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": dial tcp 10.134.0.84:8000: connect: connection refused" Nov 01 23:10:29.465703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.465673 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:29.565946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.565911 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:29.627789 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.588648 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5"] Nov 01 23:10:29.735998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.735908 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec0da7-5qgd5"] Nov 01 23:10:29.777382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.777346 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:10:29.783817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.783741 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 23:10:29.818096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.818064 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-dshm\") pod \"bf93a3f3-247b-4328-8abf-6173fb4543ec\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " Nov 01 23:10:29.818846 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.818822 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf93a3f3-247b-4328-8abf-6173fb4543ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf93a3f3-247b-4328-8abf-6173fb4543ec-tls-certs\") pod \"bf93a3f3-247b-4328-8abf-6173fb4543ec\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " Nov 01 23:10:29.819391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.819372 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-home\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-home\") pod \"bf93a3f3-247b-4328-8abf-6173fb4543ec\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " Nov 01 23:10:29.819788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.819768 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-model-cache\") pod \"bf93a3f3-247b-4328-8abf-6173fb4543ec\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " Nov 01 23:10:29.819992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.819974 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c81b530-835e-4ec1-aacb-b05d5b8efc37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c81b530-835e-4ec1-aacb-b05d5b8efc37-plugins-config-volume\") pod \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " Nov 01 23:10:29.820295 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.820277 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf93a3f3-247b-4328-8abf-6173fb4543ec-kube-api-access-p5zmq\" (UniqueName: \"kubernetes.io/projected/bf93a3f3-247b-4328-8abf-6173fb4543ec-kube-api-access-p5zmq\") pod \"bf93a3f3-247b-4328-8abf-6173fb4543ec\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " Nov 01 23:10:29.820445 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.820430 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c81b530-835e-4ec1-aacb-b05d5b8efc37-kube-api-access-8rwr2\" (UniqueName: \"kubernetes.io/projected/6c81b530-835e-4ec1-aacb-b05d5b8efc37-kube-api-access-8rwr2\") pod \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\" (UID: \"6c81b530-835e-4ec1-aacb-b05d5b8efc37\") " Nov 01 23:10:29.820597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.820560 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-tmp-dir\") pod \"bf93a3f3-247b-4328-8abf-6173fb4543ec\" (UID: \"bf93a3f3-247b-4328-8abf-6173fb4543ec\") " Nov 01 23:10:29.822221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.819053 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:29.822221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.819702 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-home" pod "bf93a3f3-247b-4328-8abf-6173fb4543ec" (UID: "bf93a3f3-247b-4328-8abf-6173fb4543ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.822221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.820482 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-model-cache" pod "bf93a3f3-247b-4328-8abf-6173fb4543ec" (UID: "bf93a3f3-247b-4328-8abf-6173fb4543ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.822221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.820560 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c81b530-835e-4ec1-aacb-b05d5b8efc37-plugins-config-volume" pod "6c81b530-835e-4ec1-aacb-b05d5b8efc37" (UID: "6c81b530-835e-4ec1-aacb-b05d5b8efc37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:29.822221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.821022 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-tmp-dir" pod "bf93a3f3-247b-4328-8abf-6173fb4543ec" (UID: "bf93a3f3-247b-4328-8abf-6173fb4543ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.822221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.821435 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.823021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.821455 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.823021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.822753 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.823021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.822772 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c81b530-835e-4ec1-aacb-b05d5b8efc37-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.823557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.823524 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf93a3f3-247b-4328-8abf-6173fb4543ec-tls-certs" pod "bf93a3f3-247b-4328-8abf-6173fb4543ec" (UID: "bf93a3f3-247b-4328-8abf-6173fb4543ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:29.824618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.824546 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c81b530-835e-4ec1-aacb-b05d5b8efc37-kube-api-access-8rwr2" pod "6c81b530-835e-4ec1-aacb-b05d5b8efc37" (UID: "6c81b530-835e-4ec1-aacb-b05d5b8efc37"). InnerVolumeSpecName "kube-api-access-8rwr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:29.827119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.827096 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-dshm" pod "bf93a3f3-247b-4328-8abf-6173fb4543ec" (UID: "bf93a3f3-247b-4328-8abf-6173fb4543ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:29.829764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.829742 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf93a3f3-247b-4328-8abf-6173fb4543ec-kube-api-access-p5zmq" pod "bf93a3f3-247b-4328-8abf-6173fb4543ec" (UID: "bf93a3f3-247b-4328-8abf-6173fb4543ec"). InnerVolumeSpecName "kube-api-access-p5zmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:29.841717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.841691 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:10:29.923651 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.923615 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-plugins-config-volume\") pod \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " Nov 01 23:10:29.923900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.923785 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-kube-api-access-pgz6h\" (UniqueName: \"kubernetes.io/projected/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-kube-api-access-pgz6h\") pod \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\" (UID: \"9edf3a9a-a7ca-4a51-ba63-eb40609e5357\") " Nov 01 23:10:29.924062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.924016 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-plugins-config-volume" pod "9edf3a9a-a7ca-4a51-ba63-eb40609e5357" (UID: "9edf3a9a-a7ca-4a51-ba63-eb40609e5357"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:29.924555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.924529 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf93a3f3-247b-4328-8abf-6173fb4543ec-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.924683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.924561 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.924683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.924592 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf93a3f3-247b-4328-8abf-6173fb4543ec-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.924683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.924610 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5zmq\" (UniqueName: \"kubernetes.io/projected/bf93a3f3-247b-4328-8abf-6173fb4543ec-kube-api-access-p5zmq\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.924683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.924622 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwr2\" (UniqueName: \"kubernetes.io/projected/6c81b530-835e-4ec1-aacb-b05d5b8efc37-kube-api-access-8rwr2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:29.926093 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.926067 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-kube-api-access-pgz6h" pod "9edf3a9a-a7ca-4a51-ba63-eb40609e5357" (UID: "9edf3a9a-a7ca-4a51-ba63-eb40609e5357"). InnerVolumeSpecName "kube-api-access-pgz6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:29.929460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.929433 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" event={"ID":"cc1f7468-8561-40a0-ae39-70aafe1a7cb8","Type":"ContainerStarted","Data":"e5095be26cf04e62a4d0873d9a191590fd214f043911b0d582eeed17ee7d38e0"} Nov 01 23:10:29.929566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.929485 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" event={"ID":"cc1f7468-8561-40a0-ae39-70aafe1a7cb8","Type":"ContainerStarted","Data":"c59c99ca9267269d40e5600ac67d49ed7c949749795440054d5406f7f24454c5"} Nov 01 23:10:29.930942 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.930920 4981 generic.go:356] "Generic (PLEG): container finished" podID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerID="38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d" exitCode=0 Nov 01 23:10:29.931018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.930956 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" Nov 01 23:10:29.931018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.930996 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" event={"ID":"bf93a3f3-247b-4328-8abf-6173fb4543ec","Type":"ContainerDied","Data":"38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d"} Nov 01 23:10:29.931105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.931032 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72" event={"ID":"bf93a3f3-247b-4328-8abf-6173fb4543ec","Type":"ContainerDied","Data":"e83c443097b3781e0966c461c4b76dbc5e58129d7888c790a8d68472ea5dc107"} Nov 01 23:10:29.931105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.931053 4981 scope.go:122] "RemoveContainer" containerID="38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d" Nov 01 23:10:29.932407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.932388 4981 generic.go:356] "Generic (PLEG): container finished" podID="9edf3a9a-a7ca-4a51-ba63-eb40609e5357" containerID="bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07" exitCode=0 Nov 01 23:10:29.932506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.932480 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" event={"ID":"9edf3a9a-a7ca-4a51-ba63-eb40609e5357","Type":"ContainerDied","Data":"bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07"} Nov 01 23:10:29.932587 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.932513 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" event={"ID":"9edf3a9a-a7ca-4a51-ba63-eb40609e5357","Type":"ContainerDied","Data":"0d81979d6ca3aba8bafe087f305df6635acd42e393f8a4dea055452142d143e3"} Nov 01 23:10:29.932587 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.932555 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw" Nov 01 23:10:29.937426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.937317 4981 generic.go:356] "Generic (PLEG): container finished" podID="6c81b530-835e-4ec1-aacb-b05d5b8efc37" containerID="92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5" exitCode=0 Nov 01 23:10:29.937426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.937346 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" event={"ID":"6c81b530-835e-4ec1-aacb-b05d5b8efc37","Type":"ContainerDied","Data":"92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5"} Nov 01 23:10:29.937426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.937370 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" Nov 01 23:10:29.937426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.937396 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8" event={"ID":"6c81b530-835e-4ec1-aacb-b05d5b8efc37","Type":"ContainerDied","Data":"6f1af79da688ba65b26348d4b4b88c6cee3054c6545b2dae21ca06798cbd97a3"} Nov 01 23:10:29.939253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.939224 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" event={"ID":"da815bca-be05-447f-80a2-90ace52e4e04","Type":"ContainerStarted","Data":"f836dc55129f503f13f79fd2cadb147cbc11ba2a485154179d7bb711a85135da"} Nov 01 23:10:29.939338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.939272 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" event={"ID":"da815bca-be05-447f-80a2-90ace52e4e04","Type":"ContainerStarted","Data":"1922ad8eb8a5b088093f8911a437098c7e38c6302a86b2ce17a53fd302590531"} Nov 01 23:10:29.942515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.942493 4981 scope.go:122] "RemoveContainer" containerID="38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d" Nov 01 23:10:29.942860 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:29.942834 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d\": container with ID starting with 38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d not found: ID does not exist" containerID="38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d" Nov 01 23:10:29.942953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.942869 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d"} err="failed to get container status \"38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d\": rpc error: code = NotFound desc = could not find container \"38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d\": container with ID starting with 38bd503c7c79a360155bdb2eb2365baa6adad49c9614cbce4df3d14c2e03447d not found: ID does not exist" Nov 01 23:10:29.942953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.942889 4981 scope.go:122] "RemoveContainer" containerID="bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07" Nov 01 23:10:29.952324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.952285 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" podStartSLOduration=1.95227297 podStartE2EDuration="1.95227297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:29.948185173 +0000 UTC m=+2795.376063106" watchObservedRunningTime="2026-11-01 23:10:29.95227297 +0000 UTC m=+2795.380150902" Nov 01 23:10:29.955001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.954980 4981 scope.go:122] "RemoveContainer" containerID="bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07" Nov 01 23:10:29.955262 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:29.955241 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07\": container with ID starting with bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07 not found: ID does not exist" containerID="bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07" Nov 01 23:10:29.955323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.955267 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07"} err="failed to get container status \"bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07\": rpc error: code = NotFound desc = could not find container \"bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07\": container with ID starting with bff23b66d78608f71c4dc1dc5a4eb4db7ccebd94601c7ef9739db60f4bb3af07 not found: ID does not exist" Nov 01 23:10:29.955323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.955282 4981 scope.go:122] "RemoveContainer" containerID="92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5" Nov 01 23:10:29.965603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.965584 4981 scope.go:122] "RemoveContainer" containerID="92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5" Nov 01 23:10:29.965869 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:29.965850 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5\": container with ID starting with 92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5 not found: ID does not exist" containerID="92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5" Nov 01 23:10:29.965936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.965878 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5"} err="failed to get container status \"92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5\": rpc error: code = NotFound desc = could not find container \"92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5\": container with ID starting with 92c63c73ac97ac5cf48932f8730004f7368a40382fb3d091db13bb29837cd6f5 not found: ID does not exist" Nov 01 23:10:29.967981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.967960 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72"] Nov 01 23:10:29.972871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.972852 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9ad303/distinct-w2-9ad303-kserve-585b6c6d77-8tp72"] Nov 01 23:10:29.979190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.979161 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:29.987896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:29.985114 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" podStartSLOduration=1.985098959 podStartE2EDuration="1.985098959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:29.98440261 +0000 UTC m=+2795.412280544" watchObservedRunningTime="2026-11-01 23:10:29.985098959 +0000 UTC m=+2795.412976886" Nov 01 23:10:30.006733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.006694 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw"] Nov 01 23:10:30.018293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.017496 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec0da7-5bd85dc767-4pwmw"] Nov 01 23:10:30.027929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.027893 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8"] Nov 01 23:10:30.029459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.029428 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgz6h\" (UniqueName: \"kubernetes.io/projected/9edf3a9a-a7ca-4a51-ba63-eb40609e5357-kube-api-access-pgz6h\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:30.033857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.033817 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec0da7-694ddb46fb-p5hp8"] Nov 01 23:10:30.537285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.537253 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb"] Nov 01 23:10:30.537992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.537971 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" Nov 01 23:10:30.537992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.537988 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" Nov 01 23:10:30.538122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.538010 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6c81b530-835e-4ec1-aacb-b05d5b8efc37" containerName="payload-processing" Nov 01 23:10:30.538122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.538016 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c81b530-835e-4ec1-aacb-b05d5b8efc37" containerName="payload-processing" Nov 01 23:10:30.538122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.538028 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9edf3a9a-a7ca-4a51-ba63-eb40609e5357" containerName="payload-pre-processing" Nov 01 23:10:30.538122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.538034 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edf3a9a-a7ca-4a51-ba63-eb40609e5357" containerName="payload-pre-processing" Nov 01 23:10:30.538260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.538146 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c81b530-835e-4ec1-aacb-b05d5b8efc37" containerName="payload-processing" Nov 01 23:10:30.538260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.538157 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edf3a9a-a7ca-4a51-ba63-eb40609e5357" containerName="payload-pre-processing" Nov 01 23:10:30.538260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.538170 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" containerName="main" Nov 01 23:10:30.542173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.542157 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.554023 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.553996 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb"] Nov 01 23:10:30.639269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.639235 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmtp\" (UniqueName: \"kubernetes.io/projected/a3baea25-dcda-4716-9176-538077fedfa1-kube-api-access-xrmtp\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.639459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.639445 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.639505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.639473 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.653693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.653669 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:30.741383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.741287 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.741383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.741352 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.741640 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.741461 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmtp\" (UniqueName: \"kubernetes.io/projected/a3baea25-dcda-4716-9176-538077fedfa1-kube-api-access-xrmtp\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.742784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.742758 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.744827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.744780 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0257d23e-b180-43b1-bfed-5a499fc4fb63" path="/var/lib/kubelet/pods/0257d23e-b180-43b1-bfed-5a499fc4fb63/volumes" Nov 01 23:10:30.745236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.745210 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.745379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.745357 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c81b530-835e-4ec1-aacb-b05d5b8efc37" path="/var/lib/kubelet/pods/6c81b530-835e-4ec1-aacb-b05d5b8efc37/volumes" Nov 01 23:10:30.745969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.745949 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edf3a9a-a7ca-4a51-ba63-eb40609e5357" path="/var/lib/kubelet/pods/9edf3a9a-a7ca-4a51-ba63-eb40609e5357/volumes" Nov 01 23:10:30.747311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.747288 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf93a3f3-247b-4328-8abf-6173fb4543ec" path="/var/lib/kubelet/pods/bf93a3f3-247b-4328-8abf-6173fb4543ec/volumes" Nov 01 23:10:30.749981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.749963 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmtp\" (UniqueName: \"kubernetes.io/projected/a3baea25-dcda-4716-9176-538077fedfa1-kube-api-access-xrmtp\") pod \"maas-api-revoke-sub-8d6eed9c-sf4fb\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.860347 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.860321 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:30.954335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.954302 4981 generic.go:356] "Generic (PLEG): container finished" podID="da815bca-be05-447f-80a2-90ace52e4e04" containerID="f836dc55129f503f13f79fd2cadb147cbc11ba2a485154179d7bb711a85135da" exitCode=0 Nov 01 23:10:30.954495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.954382 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" event={"ID":"da815bca-be05-447f-80a2-90ace52e4e04","Type":"ContainerDied","Data":"f836dc55129f503f13f79fd2cadb147cbc11ba2a485154179d7bb711a85135da"} Nov 01 23:10:30.956277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.956252 4981 generic.go:356] "Generic (PLEG): container finished" podID="cc1f7468-8561-40a0-ae39-70aafe1a7cb8" containerID="e5095be26cf04e62a4d0873d9a191590fd214f043911b0d582eeed17ee7d38e0" exitCode=0 Nov 01 23:10:30.957319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:30.956351 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" event={"ID":"cc1f7468-8561-40a0-ae39-70aafe1a7cb8","Type":"ContainerDied","Data":"e5095be26cf04e62a4d0873d9a191590fd214f043911b0d582eeed17ee7d38e0"} Nov 01 23:10:31.169907 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:31.169878 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3baea25_dcda_4716_9176_538077fedfa1.slice/crio-fb38044486496d55064ebc4a953bbf19531fd54b300f11ac0fa32adb7fe2362f WatchSource:0}: Error finding container fb38044486496d55064ebc4a953bbf19531fd54b300f11ac0fa32adb7fe2362f: Status 404 returned error can't find the container with id fb38044486496d55064ebc4a953bbf19531fd54b300f11ac0fa32adb7fe2362f Nov 01 23:10:31.171196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:31.171169 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb"] Nov 01 23:10:31.490864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:31.490843 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:31.969351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:31.969310 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" event={"ID":"a3baea25-dcda-4716-9176-538077fedfa1","Type":"ContainerStarted","Data":"d064c1c3f05b784e7d428179e37ff2a4086e2b8677880a0fc68f611ad61f2978"} Nov 01 23:10:31.969851 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:31.969375 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" event={"ID":"a3baea25-dcda-4716-9176-538077fedfa1","Type":"ContainerStarted","Data":"fb38044486496d55064ebc4a953bbf19531fd54b300f11ac0fa32adb7fe2362f"} Nov 01 23:10:32.175601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.175556 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:32.179200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.179170 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:32.196627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.196562 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" podStartSLOduration=2.196548746 podStartE2EDuration="2.196548746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:31.990147251 +0000 UTC m=+2797.418025192" watchObservedRunningTime="2026-11-01 23:10:32.196548746 +0000 UTC m=+2797.624426676" Nov 01 23:10:32.235279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.235192 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:32.282036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.271545 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-kube-api-access-tnfcg\" (UniqueName: \"kubernetes.io/projected/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-kube-api-access-tnfcg\") pod \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " Nov 01 23:10:32.282036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.272230 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-xks-ca\") pod \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " Nov 01 23:10:32.282036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.272364 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-xks-ca\") pod \"da815bca-be05-447f-80a2-90ace52e4e04\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " Nov 01 23:10:32.288751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.286735 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-kube-api-access-tnfcg" pod "cc1f7468-8561-40a0-ae39-70aafe1a7cb8" (UID: "cc1f7468-8561-40a0-ae39-70aafe1a7cb8"). InnerVolumeSpecName "kube-api-access-tnfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:32.297615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.295153 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-openshift-ca" pod "da815bca-be05-447f-80a2-90ace52e4e04" (UID: "da815bca-be05-447f-80a2-90ace52e4e04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:32.313602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.310762 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-openshift-ca\") pod \"da815bca-be05-447f-80a2-90ace52e4e04\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " Nov 01 23:10:32.313602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.310912 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da815bca-be05-447f-80a2-90ace52e4e04-kube-api-access-b665k\" (UniqueName: \"kubernetes.io/projected/da815bca-be05-447f-80a2-90ace52e4e04-kube-api-access-b665k\") pod \"da815bca-be05-447f-80a2-90ace52e4e04\" (UID: \"da815bca-be05-447f-80a2-90ace52e4e04\") " Nov 01 23:10:32.313602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.311016 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-openshift-ca\") pod \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\" (UID: \"cc1f7468-8561-40a0-ae39-70aafe1a7cb8\") " Nov 01 23:10:32.313602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.311678 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-openshift-ca" pod "cc1f7468-8561-40a0-ae39-70aafe1a7cb8" (UID: "cc1f7468-8561-40a0-ae39-70aafe1a7cb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:32.313602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.311908 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:32.313602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.311930 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnfcg\" (UniqueName: \"kubernetes.io/projected/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-kube-api-access-tnfcg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:32.313602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.311945 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:32.316752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.316563 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-xks-ca" pod "cc1f7468-8561-40a0-ae39-70aafe1a7cb8" (UID: "cc1f7468-8561-40a0-ae39-70aafe1a7cb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:32.316752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.316675 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-xks-ca" pod "da815bca-be05-447f-80a2-90ace52e4e04" (UID: "da815bca-be05-447f-80a2-90ace52e4e04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:32.318011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.317986 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da815bca-be05-447f-80a2-90ace52e4e04-kube-api-access-b665k" pod "da815bca-be05-447f-80a2-90ace52e4e04" (UID: "da815bca-be05-447f-80a2-90ace52e4e04"). InnerVolumeSpecName "kube-api-access-b665k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:32.413403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.413376 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1f7468-8561-40a0-ae39-70aafe1a7cb8-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:32.413403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.413401 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da815bca-be05-447f-80a2-90ace52e4e04-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:32.413569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.413410 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b665k\" (UniqueName: \"kubernetes.io/projected/da815bca-be05-447f-80a2-90ace52e4e04-kube-api-access-b665k\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:32.963783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.963754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:32.978133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.978106 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" Nov 01 23:10:32.978526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.978135 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj" event={"ID":"cc1f7468-8561-40a0-ae39-70aafe1a7cb8","Type":"ContainerDied","Data":"c59c99ca9267269d40e5600ac67d49ed7c949749795440054d5406f7f24454c5"} Nov 01 23:10:32.978526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.978179 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59c99ca9267269d40e5600ac67d49ed7c949749795440054d5406f7f24454c5" Nov 01 23:10:32.979911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.979886 4981 generic.go:356] "Generic (PLEG): container finished" podID="a3baea25-dcda-4716-9176-538077fedfa1" containerID="d064c1c3f05b784e7d428179e37ff2a4086e2b8677880a0fc68f611ad61f2978" exitCode=0 Nov 01 23:10:32.980046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.979973 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" event={"ID":"a3baea25-dcda-4716-9176-538077fedfa1","Type":"ContainerDied","Data":"d064c1c3f05b784e7d428179e37ff2a4086e2b8677880a0fc68f611ad61f2978"} Nov 01 23:10:32.981712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.981690 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" event={"ID":"da815bca-be05-447f-80a2-90ace52e4e04","Type":"ContainerDied","Data":"1922ad8eb8a5b088093f8911a437098c7e38c6302a86b2ce17a53fd302590531"} Nov 01 23:10:32.981840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.981726 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf" Nov 01 23:10:32.981840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:32.981730 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1922ad8eb8a5b088093f8911a437098c7e38c6302a86b2ce17a53fd302590531" Nov 01 23:10:33.630845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:33.630809 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:33.698228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:33.697100 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" podUID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:33.698228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:33.697903 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" podUID="babb7476-eab9-41b5-9729-c94dd542ad9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:34.162650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.162613 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:34.243474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.243400 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:34.244491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.244471 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-openshift-ca\") pod \"a3baea25-dcda-4716-9176-538077fedfa1\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " Nov 01 23:10:34.244689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.244673 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3baea25-dcda-4716-9176-538077fedfa1-kube-api-access-xrmtp\" (UniqueName: \"kubernetes.io/projected/a3baea25-dcda-4716-9176-538077fedfa1-kube-api-access-xrmtp\") pod \"a3baea25-dcda-4716-9176-538077fedfa1\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " Nov 01 23:10:34.244745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.244714 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-xks-ca\") pod \"a3baea25-dcda-4716-9176-538077fedfa1\" (UID: \"a3baea25-dcda-4716-9176-538077fedfa1\") " Nov 01 23:10:34.244838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.244807 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-openshift-ca" pod "a3baea25-dcda-4716-9176-538077fedfa1" (UID: "a3baea25-dcda-4716-9176-538077fedfa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:34.245241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.245217 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:34.246777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.246753 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-xks-ca" pod "a3baea25-dcda-4716-9176-538077fedfa1" (UID: "a3baea25-dcda-4716-9176-538077fedfa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:34.246883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.246773 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3baea25-dcda-4716-9176-538077fedfa1-kube-api-access-xrmtp" pod "a3baea25-dcda-4716-9176-538077fedfa1" (UID: "a3baea25-dcda-4716-9176-538077fedfa1"). InnerVolumeSpecName "kube-api-access-xrmtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:34.346968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.346936 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmtp\" (UniqueName: \"kubernetes.io/projected/a3baea25-dcda-4716-9176-538077fedfa1-kube-api-access-xrmtp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:34.346968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.346968 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3baea25-dcda-4716-9176-538077fedfa1-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:34.883864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.883788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:34.999398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.999315 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" Nov 01 23:10:34.999398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.999353 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb" event={"ID":"a3baea25-dcda-4716-9176-538077fedfa1","Type":"ContainerDied","Data":"fb38044486496d55064ebc4a953bbf19531fd54b300f11ac0fa32adb7fe2362f"} Nov 01 23:10:34.999707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:34.999400 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb38044486496d55064ebc4a953bbf19531fd54b300f11ac0fa32adb7fe2362f" Nov 01 23:10:35.663675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:35.663639 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:36.247272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.247242 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:36.350374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.350334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:36.447379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.447355 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:36.560745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.560664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:36.997683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.997653 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44"] Nov 01 23:10:36.998338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998320 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cc1f7468-8561-40a0-ae39-70aafe1a7cb8" containerName="revoke-keys" Nov 01 23:10:36.998422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998339 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1f7468-8561-40a0-ae39-70aafe1a7cb8" containerName="revoke-keys" Nov 01 23:10:36.998422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998357 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a3baea25-dcda-4716-9176-538077fedfa1" containerName="revoke-keys" Nov 01 23:10:36.998422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998362 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3baea25-dcda-4716-9176-538077fedfa1" containerName="revoke-keys" Nov 01 23:10:36.998422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998397 4981 cpu_manager.go:414] "containerMap: removing container" podUID="da815bca-be05-447f-80a2-90ace52e4e04" containerName="revoke-keys" Nov 01 23:10:36.998422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998402 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da815bca-be05-447f-80a2-90ace52e4e04" containerName="revoke-keys" Nov 01 23:10:36.998639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998499 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1f7468-8561-40a0-ae39-70aafe1a7cb8" containerName="revoke-keys" Nov 01 23:10:36.998639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998508 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3baea25-dcda-4716-9176-538077fedfa1" containerName="revoke-keys" Nov 01 23:10:36.998639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:36.998526 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="da815bca-be05-447f-80a2-90ace52e4e04" containerName="revoke-keys" Nov 01 23:10:37.002404 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.002387 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.007731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.007710 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:10:37.007887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.007827 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:10:37.009884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.009855 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44"] Nov 01 23:10:37.085336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.085303 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.085336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.085339 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.085555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.085461 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxnjj\" (UniqueName: \"kubernetes.io/projected/8cafb16a-3baa-4f83-ac08-34d356ed7c15-kube-api-access-rxnjj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.186757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.186728 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.186922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.186762 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.186922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.186858 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnjj\" (UniqueName: \"kubernetes.io/projected/8cafb16a-3baa-4f83-ac08-34d356ed7c15-kube-api-access-rxnjj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.187416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.187397 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.189071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.189052 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.195656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.195634 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnjj\" (UniqueName: \"kubernetes.io/projected/8cafb16a-3baa-4f83-ac08-34d356ed7c15-kube-api-access-rxnjj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.268482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.268400 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:37.326286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.325995 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:37.638172 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:37.638139 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cafb16a_3baa_4f83_ac08_34d356ed7c15.slice/crio-228cd4dd85f1e8cc85d14c06a440cba6f231d4c3e249b119d123a336ceaff293 WatchSource:0}: Error finding container 228cd4dd85f1e8cc85d14c06a440cba6f231d4c3e249b119d123a336ceaff293: Status 404 returned error can't find the container with id 228cd4dd85f1e8cc85d14c06a440cba6f231d4c3e249b119d123a336ceaff293 Nov 01 23:10:37.638861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:37.638834 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44"] Nov 01 23:10:38.028732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.028691 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" event={"ID":"8cafb16a-3baa-4f83-ac08-34d356ed7c15","Type":"ContainerStarted","Data":"fadc87331c5b7b30c1193f9d22494b6e085ca06fff45b1b6bc12b776b2ea9d3c"} Nov 01 23:10:38.029146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.028749 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" event={"ID":"8cafb16a-3baa-4f83-ac08-34d356ed7c15","Type":"ContainerStarted","Data":"228cd4dd85f1e8cc85d14c06a440cba6f231d4c3e249b119d123a336ceaff293"} Nov 01 23:10:38.047790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.047743 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" podStartSLOduration=2.047730215 podStartE2EDuration="2.047730215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:38.046511888 +0000 UTC m=+2803.474389821" watchObservedRunningTime="2026-11-01 23:10:38.047730215 +0000 UTC m=+2803.475608145" Nov 01 23:10:38.069108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.069081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:38.600225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.600114 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v"] Nov 01 23:10:38.600797 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.600759 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" podUID="7d12fbc1-585d-4f93-9d26-073c87434b14" containerName="maas-api" containerID="cri-o://99baf992d9a64869e780a49b4242e8f0a2c735ff28360f43e68beff8a2b261dd" gracePeriod=30 Nov 01 23:10:38.649538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.649510 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:38.685225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.685199 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c"] Nov 01 23:10:38.685406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.685383 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" podUID="a90bbf83-c043-4699-a8b1-22c63a1b1894" containerName="payload-processing" containerID="cri-o://b3a8d9aaf95b1a7d000671d74739a0a68ff9faf09a67cdfc17d9e9ef31e57bfa" gracePeriod=30 Nov 01 23:10:38.744845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.744814 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km"] Nov 01 23:10:38.745068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.745045 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" podUID="b767b8a3-2e0c-464c-b56f-b165274f06c3" containerName="payload-pre-processing" containerID="cri-o://8ff289f6a766f68364d5bd55fef255a8d8ddb4ef8fa82a0b1e8cf167c47f2c45" gracePeriod=30 Nov 01 23:10:38.767848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.767803 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:38.891613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:38.891505 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:39.071877 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.062613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:39.071877 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.066434 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8"] Nov 01 23:10:39.072407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.072373 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-01e30d-jjdt8"] Nov 01 23:10:39.075151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.075123 4981 generic.go:356] "Generic (PLEG): container finished" podID="8cafb16a-3baa-4f83-ac08-34d356ed7c15" containerID="fadc87331c5b7b30c1193f9d22494b6e085ca06fff45b1b6bc12b776b2ea9d3c" exitCode=0 Nov 01 23:10:39.075304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.075204 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" event={"ID":"8cafb16a-3baa-4f83-ac08-34d356ed7c15","Type":"ContainerDied","Data":"fadc87331c5b7b30c1193f9d22494b6e085ca06fff45b1b6bc12b776b2ea9d3c"} Nov 01 23:10:39.077424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.077404 4981 generic.go:356] "Generic (PLEG): container finished" podID="b767b8a3-2e0c-464c-b56f-b165274f06c3" containerID="8ff289f6a766f68364d5bd55fef255a8d8ddb4ef8fa82a0b1e8cf167c47f2c45" exitCode=0 Nov 01 23:10:39.077619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.077476 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" event={"ID":"b767b8a3-2e0c-464c-b56f-b165274f06c3","Type":"ContainerDied","Data":"8ff289f6a766f68364d5bd55fef255a8d8ddb4ef8fa82a0b1e8cf167c47f2c45"} Nov 01 23:10:39.080119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.080094 4981 generic.go:356] "Generic (PLEG): container finished" podID="a90bbf83-c043-4699-a8b1-22c63a1b1894" containerID="b3a8d9aaf95b1a7d000671d74739a0a68ff9faf09a67cdfc17d9e9ef31e57bfa" exitCode=0 Nov 01 23:10:39.080216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.080182 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" event={"ID":"a90bbf83-c043-4699-a8b1-22c63a1b1894","Type":"ContainerDied","Data":"b3a8d9aaf95b1a7d000671d74739a0a68ff9faf09a67cdfc17d9e9ef31e57bfa"} Nov 01 23:10:39.084752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.084728 4981 generic.go:356] "Generic (PLEG): container finished" podID="7d12fbc1-585d-4f93-9d26-073c87434b14" containerID="99baf992d9a64869e780a49b4242e8f0a2c735ff28360f43e68beff8a2b261dd" exitCode=0 Nov 01 23:10:39.084865 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.084846 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" event={"ID":"7d12fbc1-585d-4f93-9d26-073c87434b14","Type":"ContainerDied","Data":"99baf992d9a64869e780a49b4242e8f0a2c735ff28360f43e68beff8a2b261dd"} Nov 01 23:10:39.142929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.142778 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:39.146682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.146331 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:10:39.253197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.253169 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:10:39.272205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.272177 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d12fbc1-585d-4f93-9d26-073c87434b14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d12fbc1-585d-4f93-9d26-073c87434b14-openshift-service-ca\") pod \"7d12fbc1-585d-4f93-9d26-073c87434b14\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " Nov 01 23:10:39.272372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.272249 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-metrics-certs\") pod \"7d12fbc1-585d-4f93-9d26-073c87434b14\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " Nov 01 23:10:39.272372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.272297 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d12fbc1-585d-4f93-9d26-073c87434b14-kube-api-access-w4vb2\" (UniqueName: \"kubernetes.io/projected/7d12fbc1-585d-4f93-9d26-073c87434b14-kube-api-access-w4vb2\") pod \"7d12fbc1-585d-4f93-9d26-073c87434b14\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " Nov 01 23:10:39.272499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.272405 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-maas-api-tls\") pod \"7d12fbc1-585d-4f93-9d26-073c87434b14\" (UID: \"7d12fbc1-585d-4f93-9d26-073c87434b14\") " Nov 01 23:10:39.273798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.273770 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d12fbc1-585d-4f93-9d26-073c87434b14-openshift-service-ca" pod "7d12fbc1-585d-4f93-9d26-073c87434b14" (UID: "7d12fbc1-585d-4f93-9d26-073c87434b14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:39.283835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.283779 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-maas-api-tls" pod "7d12fbc1-585d-4f93-9d26-073c87434b14" (UID: "7d12fbc1-585d-4f93-9d26-073c87434b14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:39.290185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.290136 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-metrics-certs" pod "7d12fbc1-585d-4f93-9d26-073c87434b14" (UID: "7d12fbc1-585d-4f93-9d26-073c87434b14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:39.290776 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.290745 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d12fbc1-585d-4f93-9d26-073c87434b14-kube-api-access-w4vb2" pod "7d12fbc1-585d-4f93-9d26-073c87434b14" (UID: "7d12fbc1-585d-4f93-9d26-073c87434b14"). InnerVolumeSpecName "kube-api-access-w4vb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:39.325351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.325301 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:10:39.374031 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374002 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90bbf83-c043-4699-a8b1-22c63a1b1894-kube-api-access-mx488\" (UniqueName: \"kubernetes.io/projected/a90bbf83-c043-4699-a8b1-22c63a1b1894-kube-api-access-mx488\") pod \"a90bbf83-c043-4699-a8b1-22c63a1b1894\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " Nov 01 23:10:39.374204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374067 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b767b8a3-2e0c-464c-b56f-b165274f06c3-kube-api-access-nrspp\" (UniqueName: \"kubernetes.io/projected/b767b8a3-2e0c-464c-b56f-b165274f06c3-kube-api-access-nrspp\") pod \"b767b8a3-2e0c-464c-b56f-b165274f06c3\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " Nov 01 23:10:39.374204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374115 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a90bbf83-c043-4699-a8b1-22c63a1b1894-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90bbf83-c043-4699-a8b1-22c63a1b1894-plugins-config-volume\") pod \"a90bbf83-c043-4699-a8b1-22c63a1b1894\" (UID: \"a90bbf83-c043-4699-a8b1-22c63a1b1894\") " Nov 01 23:10:39.374326 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374235 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b767b8a3-2e0c-464c-b56f-b165274f06c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b767b8a3-2e0c-464c-b56f-b165274f06c3-plugins-config-volume\") pod \"b767b8a3-2e0c-464c-b56f-b165274f06c3\" (UID: \"b767b8a3-2e0c-464c-b56f-b165274f06c3\") " Nov 01 23:10:39.374705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374668 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b767b8a3-2e0c-464c-b56f-b165274f06c3-plugins-config-volume" pod "b767b8a3-2e0c-464c-b56f-b165274f06c3" (UID: "b767b8a3-2e0c-464c-b56f-b165274f06c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:39.374932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374910 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d12fbc1-585d-4f93-9d26-073c87434b14-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.375044 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374935 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.375044 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374946 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4vb2\" (UniqueName: \"kubernetes.io/projected/7d12fbc1-585d-4f93-9d26-073c87434b14-kube-api-access-w4vb2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.375044 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374955 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d12fbc1-585d-4f93-9d26-073c87434b14-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.375044 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.374964 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b767b8a3-2e0c-464c-b56f-b165274f06c3-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.375678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.375650 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90bbf83-c043-4699-a8b1-22c63a1b1894-plugins-config-volume" pod "a90bbf83-c043-4699-a8b1-22c63a1b1894" (UID: "a90bbf83-c043-4699-a8b1-22c63a1b1894"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:39.376740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.376722 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b767b8a3-2e0c-464c-b56f-b165274f06c3-kube-api-access-nrspp" pod "b767b8a3-2e0c-464c-b56f-b165274f06c3" (UID: "b767b8a3-2e0c-464c-b56f-b165274f06c3"). InnerVolumeSpecName "kube-api-access-nrspp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:39.376818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.376791 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90bbf83-c043-4699-a8b1-22c63a1b1894-kube-api-access-mx488" pod "a90bbf83-c043-4699-a8b1-22c63a1b1894" (UID: "a90bbf83-c043-4699-a8b1-22c63a1b1894"). InnerVolumeSpecName "kube-api-access-mx488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:39.476412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.476382 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx488\" (UniqueName: \"kubernetes.io/projected/a90bbf83-c043-4699-a8b1-22c63a1b1894-kube-api-access-mx488\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.476412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.476411 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrspp\" (UniqueName: \"kubernetes.io/projected/b767b8a3-2e0c-464c-b56f-b165274f06c3-kube-api-access-nrspp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.476412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.476421 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90bbf83-c043-4699-a8b1-22c63a1b1894-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:39.520571 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.520521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:39.811814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.811777 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:39.971190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:39.971155 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:40.069813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.069728 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:40.090486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.085957 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc"] Nov 01 23:10:40.097109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.097063 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9ad303-pb5bc"] Nov 01 23:10:40.107959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.107915 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" event={"ID":"a90bbf83-c043-4699-a8b1-22c63a1b1894","Type":"ContainerDied","Data":"b6fc4c24f73843cc40ca48ba5ed3638a6128600d84192a73eca19af39b0c6d8e"} Nov 01 23:10:40.108164 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.107974 4981 scope.go:122] "RemoveContainer" containerID="b3a8d9aaf95b1a7d000671d74739a0a68ff9faf09a67cdfc17d9e9ef31e57bfa" Nov 01 23:10:40.108164 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.108094 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c" Nov 01 23:10:40.116593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.113964 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" event={"ID":"7d12fbc1-585d-4f93-9d26-073c87434b14","Type":"ContainerDied","Data":"632aa355bd07317593e075c3ff021efc31702bfedfea6b6d5fd0bfd20fec1e22"} Nov 01 23:10:40.116593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.114153 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v" Nov 01 23:10:40.116804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.116732 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" Nov 01 23:10:40.116804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.116754 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km" event={"ID":"b767b8a3-2e0c-464c-b56f-b165274f06c3","Type":"ContainerDied","Data":"2515a8e0b4386588eb1db3819299e4b84d66f2f8122e567e45895a41f04c76f5"} Nov 01 23:10:40.162815 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.162782 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v"] Nov 01 23:10:40.166036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.166008 4981 scope.go:122] "RemoveContainer" containerID="99baf992d9a64869e780a49b4242e8f0a2c735ff28360f43e68beff8a2b261dd" Nov 01 23:10:40.167493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.167465 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01e30d-67864fd6c9-mr22v"] Nov 01 23:10:40.179637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.179565 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km"] Nov 01 23:10:40.182946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.182923 4981 scope.go:122] "RemoveContainer" containerID="8ff289f6a766f68364d5bd55fef255a8d8ddb4ef8fa82a0b1e8cf167c47f2c45" Nov 01 23:10:40.187396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.187358 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-01e30d-5f59ff5bbc-8g9km"] Nov 01 23:10:40.212592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.209506 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c"] Nov 01 23:10:40.215298 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.215275 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-01e30d-7856dffdd-q4c8c"] Nov 01 23:10:40.248962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.246775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:40.355930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.351173 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:40.396681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.396646 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-xks-ca\") pod \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " Nov 01 23:10:40.396882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.396776 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cafb16a-3baa-4f83-ac08-34d356ed7c15-kube-api-access-rxnjj\" (UniqueName: \"kubernetes.io/projected/8cafb16a-3baa-4f83-ac08-34d356ed7c15-kube-api-access-rxnjj\") pod \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " Nov 01 23:10:40.396942 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.396896 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-openshift-ca\") pod \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\" (UID: \"8cafb16a-3baa-4f83-ac08-34d356ed7c15\") " Nov 01 23:10:40.397305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.397277 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-openshift-ca" pod "8cafb16a-3baa-4f83-ac08-34d356ed7c15" (UID: "8cafb16a-3baa-4f83-ac08-34d356ed7c15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:40.397424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.397399 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:40.398863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.398830 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-xks-ca" pod "8cafb16a-3baa-4f83-ac08-34d356ed7c15" (UID: "8cafb16a-3baa-4f83-ac08-34d356ed7c15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:40.398975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.398912 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cafb16a-3baa-4f83-ac08-34d356ed7c15-kube-api-access-rxnjj" pod "8cafb16a-3baa-4f83-ac08-34d356ed7c15" (UID: "8cafb16a-3baa-4f83-ac08-34d356ed7c15"). InnerVolumeSpecName "kube-api-access-rxnjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:40.460240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.460200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:40.498913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.498885 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cafb16a-3baa-4f83-ac08-34d356ed7c15-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:40.498913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.498911 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxnjj\" (UniqueName: \"kubernetes.io/projected/8cafb16a-3baa-4f83-ac08-34d356ed7c15-kube-api-access-rxnjj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:40.736499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.736453 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7" path="/var/lib/kubelet/pods/43bbb3d1-e4fb-432c-9bd6-58d95b97c7b7/volumes" Nov 01 23:10:40.737072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.737054 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d12fbc1-585d-4f93-9d26-073c87434b14" path="/var/lib/kubelet/pods/7d12fbc1-585d-4f93-9d26-073c87434b14/volumes" Nov 01 23:10:40.737479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.737462 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a90bbf83-c043-4699-a8b1-22c63a1b1894" path="/var/lib/kubelet/pods/a90bbf83-c043-4699-a8b1-22c63a1b1894/volumes" Nov 01 23:10:40.738298 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.738280 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa17d216-8f94-4423-817c-fa6ea23923ce" path="/var/lib/kubelet/pods/aa17d216-8f94-4423-817c-fa6ea23923ce/volumes" Nov 01 23:10:40.738686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.738670 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b767b8a3-2e0c-464c-b56f-b165274f06c3" path="/var/lib/kubelet/pods/b767b8a3-2e0c-464c-b56f-b165274f06c3/volumes" Nov 01 23:10:40.740644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.740621 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl"] Nov 01 23:10:40.741273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741255 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8cafb16a-3baa-4f83-ac08-34d356ed7c15" containerName="revoke-keys" Nov 01 23:10:40.741365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741274 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cafb16a-3baa-4f83-ac08-34d356ed7c15" containerName="revoke-keys" Nov 01 23:10:40.741365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741291 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a90bbf83-c043-4699-a8b1-22c63a1b1894" containerName="payload-processing" Nov 01 23:10:40.741365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741299 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a90bbf83-c043-4699-a8b1-22c63a1b1894" containerName="payload-processing" Nov 01 23:10:40.741524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741362 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7d12fbc1-585d-4f93-9d26-073c87434b14" containerName="maas-api" Nov 01 23:10:40.741524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741373 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d12fbc1-585d-4f93-9d26-073c87434b14" containerName="maas-api" Nov 01 23:10:40.741524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741403 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b767b8a3-2e0c-464c-b56f-b165274f06c3" containerName="payload-pre-processing" Nov 01 23:10:40.741524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741411 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b767b8a3-2e0c-464c-b56f-b165274f06c3" containerName="payload-pre-processing" Nov 01 23:10:40.741770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741561 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d12fbc1-585d-4f93-9d26-073c87434b14" containerName="maas-api" Nov 01 23:10:40.741770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741596 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b767b8a3-2e0c-464c-b56f-b165274f06c3" containerName="payload-pre-processing" Nov 01 23:10:40.741770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741610 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cafb16a-3baa-4f83-ac08-34d356ed7c15" containerName="revoke-keys" Nov 01 23:10:40.741770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.741622 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a90bbf83-c043-4699-a8b1-22c63a1b1894" containerName="payload-processing" Nov 01 23:10:40.746721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.746704 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.753285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.753265 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl"] Nov 01 23:10:40.803246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.803217 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v222k\" (UniqueName: \"kubernetes.io/projected/77580d9e-a603-4a05-b853-1df49f18d2c2-kube-api-access-v222k\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.803422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.803366 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.803522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.803505 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.904818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.904782 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v222k\" (UniqueName: \"kubernetes.io/projected/77580d9e-a603-4a05-b853-1df49f18d2c2-kube-api-access-v222k\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.905061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.904921 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.905061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.905023 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.905670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.905643 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.907218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.907193 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:40.913138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:40.913119 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v222k\" (UniqueName: \"kubernetes.io/projected/77580d9e-a603-4a05-b853-1df49f18d2c2-kube-api-access-v222k\") pod \"maas-api-revoke-sub-419707d8-48mcl\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:41.066830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:41.066749 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:41.119908 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:41.119875 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:41.132757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:41.132050 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" Nov 01 23:10:41.132757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:41.132074 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44" event={"ID":"8cafb16a-3baa-4f83-ac08-34d356ed7c15","Type":"ContainerDied","Data":"228cd4dd85f1e8cc85d14c06a440cba6f231d4c3e249b119d123a336ceaff293"} Nov 01 23:10:41.132757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:41.132144 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228cd4dd85f1e8cc85d14c06a440cba6f231d4c3e249b119d123a336ceaff293" Nov 01 23:10:41.351021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:41.350992 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl"] Nov 01 23:10:41.352239 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:10:41.352216 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77580d9e_a603_4a05_b853_1df49f18d2c2.slice/crio-9c8a09979b7bc45466e5a321259731d3af3b8892dcf0f9448897c3121453e8c5 WatchSource:0}: Error finding container 9c8a09979b7bc45466e5a321259731d3af3b8892dcf0f9448897c3121453e8c5: Status 404 returned error can't find the container with id 9c8a09979b7bc45466e5a321259731d3af3b8892dcf0f9448897c3121453e8c5 Nov 01 23:10:41.977146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:41.977071 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:42.221419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:42.221380 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" event={"ID":"77580d9e-a603-4a05-b853-1df49f18d2c2","Type":"ContainerStarted","Data":"191947c6d92a2b468a0013dc121e63b40b55ab6b4ed1cee5f7fa97ae5794be39"} Nov 01 23:10:42.224302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:42.222160 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" event={"ID":"77580d9e-a603-4a05-b853-1df49f18d2c2","Type":"ContainerStarted","Data":"9c8a09979b7bc45466e5a321259731d3af3b8892dcf0f9448897c3121453e8c5"} Nov 01 23:10:42.252910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:42.252763 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" podStartSLOduration=2.252743745 podStartE2EDuration="2.252743745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:42.241435324 +0000 UTC m=+2807.669313258" watchObservedRunningTime="2026-11-01 23:10:42.252743745 +0000 UTC m=+2807.680621705" Nov 01 23:10:42.782154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:42.782123 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:43.230640 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:43.230607 4981 generic.go:356] "Generic (PLEG): container finished" podID="77580d9e-a603-4a05-b853-1df49f18d2c2" containerID="191947c6d92a2b468a0013dc121e63b40b55ab6b4ed1cee5f7fa97ae5794be39" exitCode=0 Nov 01 23:10:43.231029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:43.230665 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" event={"ID":"77580d9e-a603-4a05-b853-1df49f18d2c2","Type":"ContainerDied","Data":"191947c6d92a2b468a0013dc121e63b40b55ab6b4ed1cee5f7fa97ae5794be39"} Nov 01 23:10:43.678502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:43.678463 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" podUID="babb7476-eab9-41b5-9729-c94dd542ad9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:43.678684 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:43.678481 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" podUID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:10:43.791690 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:43.791655 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:44.391771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.391740 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:44.474231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.474196 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77580d9e-a603-4a05-b853-1df49f18d2c2-kube-api-access-v222k\" (UniqueName: \"kubernetes.io/projected/77580d9e-a603-4a05-b853-1df49f18d2c2-kube-api-access-v222k\") pod \"77580d9e-a603-4a05-b853-1df49f18d2c2\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " Nov 01 23:10:44.474411 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.474322 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-xks-ca\") pod \"77580d9e-a603-4a05-b853-1df49f18d2c2\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " Nov 01 23:10:44.474411 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.474360 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-openshift-ca\") pod \"77580d9e-a603-4a05-b853-1df49f18d2c2\" (UID: \"77580d9e-a603-4a05-b853-1df49f18d2c2\") " Nov 01 23:10:44.474796 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.474764 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-openshift-ca" pod "77580d9e-a603-4a05-b853-1df49f18d2c2" (UID: "77580d9e-a603-4a05-b853-1df49f18d2c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:44.476415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.476398 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-xks-ca" pod "77580d9e-a603-4a05-b853-1df49f18d2c2" (UID: "77580d9e-a603-4a05-b853-1df49f18d2c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:44.476519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.476503 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77580d9e-a603-4a05-b853-1df49f18d2c2-kube-api-access-v222k" pod "77580d9e-a603-4a05-b853-1df49f18d2c2" (UID: "77580d9e-a603-4a05-b853-1df49f18d2c2"). InnerVolumeSpecName "kube-api-access-v222k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:44.524126 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:44.524040 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:10:44.575988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.575945 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v222k\" (UniqueName: \"kubernetes.io/projected/77580d9e-a603-4a05-b853-1df49f18d2c2-kube-api-access-v222k\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.575988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.575979 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.575988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.575990 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77580d9e-a603-4a05-b853-1df49f18d2c2-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.579399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.579366 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:44.867057 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.867026 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 23:10:44.916601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.916557 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 23:10:44.980780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.980750 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-kube-api-access-8s8d6\" (UniqueName: \"kubernetes.io/projected/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-kube-api-access-8s8d6\") pod \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " Nov 01 23:10:44.980974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.980846 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babb7476-eab9-41b5-9729-c94dd542ad9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babb7476-eab9-41b5-9729-c94dd542ad9d-plugins-config-volume\") pod \"babb7476-eab9-41b5-9729-c94dd542ad9d\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " Nov 01 23:10:44.980974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.980904 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babb7476-eab9-41b5-9729-c94dd542ad9d-kube-api-access-264mj\" (UniqueName: \"kubernetes.io/projected/babb7476-eab9-41b5-9729-c94dd542ad9d-kube-api-access-264mj\") pod \"babb7476-eab9-41b5-9729-c94dd542ad9d\" (UID: \"babb7476-eab9-41b5-9729-c94dd542ad9d\") " Nov 01 23:10:44.981096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.981076 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-plugins-config-volume\") pod \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\" (UID: \"e8a95663-85ce-4cee-a4ac-b5c6ec90398c\") " Nov 01 23:10:44.981282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.981257 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babb7476-eab9-41b5-9729-c94dd542ad9d-plugins-config-volume" pod "babb7476-eab9-41b5-9729-c94dd542ad9d" (UID: "babb7476-eab9-41b5-9729-c94dd542ad9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:44.981667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.981637 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-plugins-config-volume" pod "e8a95663-85ce-4cee-a4ac-b5c6ec90398c" (UID: "e8a95663-85ce-4cee-a4ac-b5c6ec90398c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:44.981781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.981665 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/babb7476-eab9-41b5-9729-c94dd542ad9d-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:44.983325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.983296 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-kube-api-access-8s8d6" pod "e8a95663-85ce-4cee-a4ac-b5c6ec90398c" (UID: "e8a95663-85ce-4cee-a4ac-b5c6ec90398c"). InnerVolumeSpecName "kube-api-access-8s8d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:44.983406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:44.983356 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babb7476-eab9-41b5-9729-c94dd542ad9d-kube-api-access-264mj" pod "babb7476-eab9-41b5-9729-c94dd542ad9d" (UID: "babb7476-eab9-41b5-9729-c94dd542ad9d"). InnerVolumeSpecName "kube-api-access-264mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:45.082741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.082660 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8d6\" (UniqueName: \"kubernetes.io/projected/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-kube-api-access-8s8d6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:45.082741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.082688 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264mj\" (UniqueName: \"kubernetes.io/projected/babb7476-eab9-41b5-9729-c94dd542ad9d-kube-api-access-264mj\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:45.082741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.082697 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a95663-85ce-4cee-a4ac-b5c6ec90398c-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:45.248313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.248282 4981 generic.go:356] "Generic (PLEG): container finished" podID="babb7476-eab9-41b5-9729-c94dd542ad9d" containerID="d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c" exitCode=0 Nov 01 23:10:45.248461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.248332 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" Nov 01 23:10:45.248461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.248370 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" event={"ID":"babb7476-eab9-41b5-9729-c94dd542ad9d","Type":"ContainerDied","Data":"d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c"} Nov 01 23:10:45.248461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.248420 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf" event={"ID":"babb7476-eab9-41b5-9729-c94dd542ad9d","Type":"ContainerDied","Data":"8cbb318cb62f448b92f059031b606c2532f0f5efe0ce8c2050ebbb55592b3393"} Nov 01 23:10:45.248461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.248442 4981 scope.go:122] "RemoveContainer" containerID="d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c" Nov 01 23:10:45.249994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.249975 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" Nov 01 23:10:45.250106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.250005 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl" event={"ID":"77580d9e-a603-4a05-b853-1df49f18d2c2","Type":"ContainerDied","Data":"9c8a09979b7bc45466e5a321259731d3af3b8892dcf0f9448897c3121453e8c5"} Nov 01 23:10:45.250106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.250051 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c8a09979b7bc45466e5a321259731d3af3b8892dcf0f9448897c3121453e8c5" Nov 01 23:10:45.251730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.251700 4981 generic.go:356] "Generic (PLEG): container finished" podID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" containerID="f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01" exitCode=0 Nov 01 23:10:45.251886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.251744 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" Nov 01 23:10:45.251886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.251808 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" event={"ID":"e8a95663-85ce-4cee-a4ac-b5c6ec90398c","Type":"ContainerDied","Data":"f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01"} Nov 01 23:10:45.251886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.251852 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2" event={"ID":"e8a95663-85ce-4cee-a4ac-b5c6ec90398c","Type":"ContainerDied","Data":"61234577382c3779852e64bf3e42182d4eafe05d018510d39282778f66d59cee"} Nov 01 23:10:45.265106 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.265074 4981 scope.go:122] "RemoveContainer" containerID="d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c" Nov 01 23:10:45.266736 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:45.266712 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c\": container with ID starting with d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c not found: ID does not exist" containerID="d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c" Nov 01 23:10:45.266921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.266892 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c"} err="failed to get container status \"d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c\": rpc error: code = NotFound desc = could not find container \"d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c\": container with ID starting with d7475e54fb95c58152dc1234bd2a3580fd621c9f3fe4e7c38896fc3e0543827c not found: ID does not exist" Nov 01 23:10:45.266921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.266920 4981 scope.go:122] "RemoveContainer" containerID="f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01" Nov 01 23:10:45.277014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.276999 4981 scope.go:122] "RemoveContainer" containerID="f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01" Nov 01 23:10:45.277254 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:45.277235 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01\": container with ID starting with f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01 not found: ID does not exist" containerID="f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01" Nov 01 23:10:45.277319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.277260 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01"} err="failed to get container status \"f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01\": rpc error: code = NotFound desc = could not find container \"f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01\": container with ID starting with f8aa2ae023366ab3171c11fe3aae127ea947fe6691e8c5912141ac7508a20a01 not found: ID does not exist" Nov 01 23:10:45.282849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.282826 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf"] Nov 01 23:10:45.292217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.292197 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-9ad303-5bf6949899-nkkgf"] Nov 01 23:10:45.307507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.307485 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2"] Nov 01 23:10:45.311143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.311119 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9ad303-9d89c6c4c-9f9q2"] Nov 01 23:10:45.379313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:45.379285 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:46.057633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.057597 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:46.298663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.298623 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:46.399910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.399784 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:46.576724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.576687 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b"] Nov 01 23:10:46.576949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.576925 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" containerID="cri-o://30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a" gracePeriod=30 Nov 01 23:10:46.602840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.602811 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:46.776700 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.776664 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babb7476-eab9-41b5-9729-c94dd542ad9d" path="/var/lib/kubelet/pods/babb7476-eab9-41b5-9729-c94dd542ad9d/volumes" Nov 01 23:10:46.777365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.777341 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" path="/var/lib/kubelet/pods/e8a95663-85ce-4cee-a4ac-b5c6ec90398c/volumes" Nov 01 23:10:46.896645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:46.896613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:47.029641 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.029548 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb"] Nov 01 23:10:47.030335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030312 4981 cpu_manager.go:414] "containerMap: removing container" podUID="babb7476-eab9-41b5-9729-c94dd542ad9d" containerName="payload-pre-processing" Nov 01 23:10:47.030335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030335 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babb7476-eab9-41b5-9729-c94dd542ad9d" containerName="payload-pre-processing" Nov 01 23:10:47.030511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030358 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" containerName="payload-processing" Nov 01 23:10:47.030511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030365 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" containerName="payload-processing" Nov 01 23:10:47.030511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030386 4981 cpu_manager.go:414] "containerMap: removing container" podUID="77580d9e-a603-4a05-b853-1df49f18d2c2" containerName="revoke-keys" Nov 01 23:10:47.030511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030394 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77580d9e-a603-4a05-b853-1df49f18d2c2" containerName="revoke-keys" Nov 01 23:10:47.030741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030623 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="babb7476-eab9-41b5-9729-c94dd542ad9d" containerName="payload-pre-processing" Nov 01 23:10:47.030741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030641 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a95663-85ce-4cee-a4ac-b5c6ec90398c" containerName="payload-processing" Nov 01 23:10:47.030741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.030653 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="77580d9e-a603-4a05-b853-1df49f18d2c2" containerName="revoke-keys" Nov 01 23:10:47.035895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.035873 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.044596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.042882 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:10:47.044596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.042995 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:10:47.046182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.046148 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb"] Nov 01 23:10:47.067715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.064809 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:10:47.067715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.064858 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:10:47.153629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.153565 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.153805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.153789 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqd4\" (UniqueName: \"kubernetes.io/projected/541276ef-bf48-47fb-b6ac-67d81069b23e-kube-api-access-rlqd4\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.153889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.153818 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.256024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.255984 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.256187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.256156 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqd4\" (UniqueName: \"kubernetes.io/projected/541276ef-bf48-47fb-b6ac-67d81069b23e-kube-api-access-rlqd4\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.256250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.256181 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.256929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.256904 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.259237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.259210 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.271724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.271686 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqd4\" (UniqueName: \"kubernetes.io/projected/541276ef-bf48-47fb-b6ac-67d81069b23e-kube-api-access-rlqd4\") pod \"maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.282491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.282374 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:47.358967 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.358933 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:47.386598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.386550 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:47.546239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.546126 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:47.641431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.641388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:47.668802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.667966 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb"] Nov 01 23:10:47.893629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:47.893472 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:48.046599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:48.045053 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:48.065991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:48.050624 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt"] Nov 01 23:10:48.082675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:48.082644 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21387493-9mwnt"] Nov 01 23:10:48.285934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:48.285898 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" event={"ID":"541276ef-bf48-47fb-b6ac-67d81069b23e","Type":"ContainerStarted","Data":"6bf327902a76a08630be3f6926568544a341837611679b44359240a91ccd06c0"} Nov 01 23:10:48.286103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:48.285950 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" event={"ID":"541276ef-bf48-47fb-b6ac-67d81069b23e","Type":"ContainerStarted","Data":"01bd45d9af53c6337fa2020f986f3e1270a7b180fabf5514d1d747eea977d3a5"} Nov 01 23:10:48.336866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:48.336809 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" podStartSLOduration=1.336791337 podStartE2EDuration="1.336791337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:48.333214385 +0000 UTC m=+2813.761092318" watchObservedRunningTime="2026-11-01 23:10:48.336791337 +0000 UTC m=+2813.764669269" Nov 01 23:10:48.734873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:48.734842 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8" path="/var/lib/kubelet/pods/1f073c38-e91a-4d1e-bfde-1ae99f4d9fe8/volumes" Nov 01 23:10:49.104483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:49.104451 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:49.295849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:49.295812 4981 generic.go:356] "Generic (PLEG): container finished" podID="541276ef-bf48-47fb-b6ac-67d81069b23e" containerID="6bf327902a76a08630be3f6926568544a341837611679b44359240a91ccd06c0" exitCode=0 Nov 01 23:10:49.296011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:49.295901 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" event={"ID":"541276ef-bf48-47fb-b6ac-67d81069b23e","Type":"ContainerDied","Data":"6bf327902a76a08630be3f6926568544a341837611679b44359240a91ccd06c0"} Nov 01 23:10:49.758725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:49.758687 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:50.446722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.446691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:50.456796 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.456774 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:50.610438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.610345 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541276ef-bf48-47fb-b6ac-67d81069b23e-kube-api-access-rlqd4\" (UniqueName: \"kubernetes.io/projected/541276ef-bf48-47fb-b6ac-67d81069b23e-kube-api-access-rlqd4\") pod \"541276ef-bf48-47fb-b6ac-67d81069b23e\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " Nov 01 23:10:50.610654 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.610520 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-openshift-ca\") pod \"541276ef-bf48-47fb-b6ac-67d81069b23e\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " Nov 01 23:10:50.610734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.610660 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-xks-ca\") pod \"541276ef-bf48-47fb-b6ac-67d81069b23e\" (UID: \"541276ef-bf48-47fb-b6ac-67d81069b23e\") " Nov 01 23:10:50.610939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.610911 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-openshift-ca" pod "541276ef-bf48-47fb-b6ac-67d81069b23e" (UID: "541276ef-bf48-47fb-b6ac-67d81069b23e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:50.611140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.611111 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:50.612909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.612880 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541276ef-bf48-47fb-b6ac-67d81069b23e-kube-api-access-rlqd4" pod "541276ef-bf48-47fb-b6ac-67d81069b23e" (UID: "541276ef-bf48-47fb-b6ac-67d81069b23e"). InnerVolumeSpecName "kube-api-access-rlqd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:50.613142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.613123 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-xks-ca" pod "541276ef-bf48-47fb-b6ac-67d81069b23e" (UID: "541276ef-bf48-47fb-b6ac-67d81069b23e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:50.712465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.712426 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541276ef-bf48-47fb-b6ac-67d81069b23e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:50.712465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:50.712458 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqd4\" (UniqueName: \"kubernetes.io/projected/541276ef-bf48-47fb-b6ac-67d81069b23e-kube-api-access-rlqd4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:51.239460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.239416 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:51.330237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.330180 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" event={"ID":"541276ef-bf48-47fb-b6ac-67d81069b23e","Type":"ContainerDied","Data":"01bd45d9af53c6337fa2020f986f3e1270a7b180fabf5514d1d747eea977d3a5"} Nov 01 23:10:51.330457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.330245 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01bd45d9af53c6337fa2020f986f3e1270a7b180fabf5514d1d747eea977d3a5" Nov 01 23:10:51.330457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.330331 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb" Nov 01 23:10:51.347359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.347323 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9"] Nov 01 23:10:51.348362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.348335 4981 cpu_manager.go:414] "containerMap: removing container" podUID="541276ef-bf48-47fb-b6ac-67d81069b23e" containerName="revoke-keys" Nov 01 23:10:51.348362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.348362 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541276ef-bf48-47fb-b6ac-67d81069b23e" containerName="revoke-keys" Nov 01 23:10:51.348525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.348511 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="541276ef-bf48-47fb-b6ac-67d81069b23e" containerName="revoke-keys" Nov 01 23:10:51.352304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.352284 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.356362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.355943 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:10:51.356543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.356063 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:10:51.356840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.356820 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:10:51.356960 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.356909 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:10:51.361800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.361775 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9"] Nov 01 23:10:51.524272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.524189 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxks\" (UniqueName: \"kubernetes.io/projected/65153a3e-b028-4d7c-9285-cefad05006a5-kube-api-access-tqxks\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.524272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.524242 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.524706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.524357 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.625858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.625830 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxks\" (UniqueName: \"kubernetes.io/projected/65153a3e-b028-4d7c-9285-cefad05006a5-kube-api-access-tqxks\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.625973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.625875 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.625973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.625929 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.626540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.626518 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.628350 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.628330 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.633872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.633853 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxks\" (UniqueName: \"kubernetes.io/projected/65153a3e-b028-4d7c-9285-cefad05006a5-kube-api-access-tqxks\") pod \"maas-api-revoke-sub-72f93753-29ds9\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.668793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.668773 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:51.840551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.840516 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:51.901155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.901124 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:10:51.981858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:51.981817 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:52.049979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.049936 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-envoy\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050006 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-certs\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050134 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istiod-ca-cert\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050197 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-socket\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050263 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-credential-socket\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050395 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050331 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-data\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050395 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050388 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-podinfo\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050432 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-kube-api-access-4hqq6\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-kube-api-access-4hqq6\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.050492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.050467 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-token\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-token\") pod \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\" (UID: \"39d5338e-6670-4c89-adbe-bb8850cbaaa4\") " Nov 01 23:10:52.064847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.061955 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-socket" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.064847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.062369 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9"] Nov 01 23:10:52.064847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.062659 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-data" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.064847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.062880 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-credential-socket" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.064847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.064768 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-certs" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.065152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.065036 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istiod-ca-cert" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:52.066702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.066245 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-token" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.071312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.070760 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-podinfo" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:10:52.074879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.074851 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-envoy" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:10:52.080337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.080021 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-kube-api-access-4hqq6" pod "39d5338e-6670-4c89-adbe-bb8850cbaaa4" (UID: "39d5338e-6670-4c89-adbe-bb8850cbaaa4"). InnerVolumeSpecName "kube-api-access-4hqq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:52.087088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.087012 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.156878 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.156919 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.156933 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.156946 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.156961 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.156974 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.156986 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.157002 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hqq6\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-kube-api-access-4hqq6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.159836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.157015 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39d5338e-6670-4c89-adbe-bb8850cbaaa4-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:52.286112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.286086 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:52.339089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.339019 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" event={"ID":"65153a3e-b028-4d7c-9285-cefad05006a5","Type":"ContainerStarted","Data":"e2fef6539901ce53dd15d40fd1f490b68fdb1948b46ec86ea3cf6d27bcf8b872"} Nov 01 23:10:52.340763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.340733 4981 generic.go:356] "Generic (PLEG): container finished" podID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerID="30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a" exitCode=0 Nov 01 23:10:52.340913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.340789 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" Nov 01 23:10:52.340913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.340843 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" event={"ID":"39d5338e-6670-4c89-adbe-bb8850cbaaa4","Type":"ContainerDied","Data":"30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a"} Nov 01 23:10:52.341026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.340901 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b" event={"ID":"39d5338e-6670-4c89-adbe-bb8850cbaaa4","Type":"ContainerDied","Data":"67e95b954cf9ac20478768fe820d7b1d8d09fcb27d0c12fc0be8aab88924e5c5"} Nov 01 23:10:52.341026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.340947 4981 scope.go:122] "RemoveContainer" containerID="30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a" Nov 01 23:10:52.353513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.353486 4981 scope.go:122] "RemoveContainer" containerID="30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a" Nov 01 23:10:52.353886 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:10:52.353860 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a\": container with ID starting with 30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a not found: ID does not exist" containerID="30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a" Nov 01 23:10:52.353977 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.353897 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a"} err="failed to get container status \"30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a\": rpc error: code = NotFound desc = could not find container \"30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a\": container with ID starting with 30722889973fbededf253eed44ffe0fd2702416ec8d9acf776c1701ed6bfe88a not found: ID does not exist" Nov 01 23:10:52.367627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.367595 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b"] Nov 01 23:10:52.371646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.371602 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-01e30d-openshift-default-647c5746f-6tj9b"] Nov 01 23:10:52.735393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:52.735362 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" path="/var/lib/kubelet/pods/39d5338e-6670-4c89-adbe-bb8850cbaaa4/volumes" Nov 01 23:10:53.052888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:53.052860 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:53.356760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:53.356666 4981 generic.go:356] "Generic (PLEG): container finished" podID="65153a3e-b028-4d7c-9285-cefad05006a5" containerID="dedf1e800ee5d30ee80797fa21eb88bafc1b14905976102894577fa363ea3c30" exitCode=0 Nov 01 23:10:53.356760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:53.356728 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" event={"ID":"65153a3e-b028-4d7c-9285-cefad05006a5","Type":"ContainerDied","Data":"dedf1e800ee5d30ee80797fa21eb88bafc1b14905976102894577fa363ea3c30"} Nov 01 23:10:53.828947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:53.828914 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:54.515200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.515176 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:54.677892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.677865 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:54.699165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.699141 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-xks-ca\") pod \"65153a3e-b028-4d7c-9285-cefad05006a5\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " Nov 01 23:10:54.699301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.699251 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65153a3e-b028-4d7c-9285-cefad05006a5-kube-api-access-tqxks\" (UniqueName: \"kubernetes.io/projected/65153a3e-b028-4d7c-9285-cefad05006a5-kube-api-access-tqxks\") pod \"65153a3e-b028-4d7c-9285-cefad05006a5\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " Nov 01 23:10:54.699356 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.699316 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-openshift-ca\") pod \"65153a3e-b028-4d7c-9285-cefad05006a5\" (UID: \"65153a3e-b028-4d7c-9285-cefad05006a5\") " Nov 01 23:10:54.699724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.699700 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-openshift-ca" pod "65153a3e-b028-4d7c-9285-cefad05006a5" (UID: "65153a3e-b028-4d7c-9285-cefad05006a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:10:54.701255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.701231 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-xks-ca" pod "65153a3e-b028-4d7c-9285-cefad05006a5" (UID: "65153a3e-b028-4d7c-9285-cefad05006a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:10:54.701329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.701309 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65153a3e-b028-4d7c-9285-cefad05006a5-kube-api-access-tqxks" pod "65153a3e-b028-4d7c-9285-cefad05006a5" (UID: "65153a3e-b028-4d7c-9285-cefad05006a5"). InnerVolumeSpecName "kube-api-access-tqxks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:10:54.801190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.801151 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxks\" (UniqueName: \"kubernetes.io/projected/65153a3e-b028-4d7c-9285-cefad05006a5-kube-api-access-tqxks\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:54.801353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.801192 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:54.801353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:54.801208 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65153a3e-b028-4d7c-9285-cefad05006a5-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:10:55.373552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:55.373520 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" Nov 01 23:10:55.373552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:55.373546 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9" event={"ID":"65153a3e-b028-4d7c-9285-cefad05006a5","Type":"ContainerDied","Data":"e2fef6539901ce53dd15d40fd1f490b68fdb1948b46ec86ea3cf6d27bcf8b872"} Nov 01 23:10:55.373995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:55.373607 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2fef6539901ce53dd15d40fd1f490b68fdb1948b46ec86ea3cf6d27bcf8b872" Nov 01 23:10:55.567295 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:55.567259 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:56.476709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:56.476679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:57.345160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:57.345125 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:58.025639 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:58.025601 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh"] Nov 01 23:10:58.036672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:58.036636 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ec40d-kbssh"] Nov 01 23:10:58.071446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:58.071412 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:58.723984 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:58.723954 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:58.734672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:58.734645 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495deed8-b425-4e18-b6d5-6cf2f3c2b42a" path="/var/lib/kubelet/pods/495deed8-b425-4e18-b6d5-6cf2f3c2b42a/volumes" Nov 01 23:10:58.860373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:58.860332 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:58.964689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:58.964658 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:59.057234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:59.057157 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:59.094423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:59.094389 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44"] Nov 01 23:10:59.097901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:59.097875 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-c6cec6-ftg44"] Nov 01 23:10:59.161208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:59.161176 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:59.561906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:59.561873 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:10:59.987501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:10:59.987467 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:00.676340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:00.676305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:00.758424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:00.758308 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cafb16a-3baa-4f83-ac08-34d356ed7c15" path="/var/lib/kubelet/pods/8cafb16a-3baa-4f83-ac08-34d356ed7c15/volumes" Nov 01 23:11:01.350345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.350315 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:01.699954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.699925 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk"] Nov 01 23:11:01.700559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.700543 4981 cpu_manager.go:414] "containerMap: removing container" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" Nov 01 23:11:01.700626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.700560 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" Nov 01 23:11:01.700626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.700591 4981 cpu_manager.go:414] "containerMap: removing container" podUID="65153a3e-b028-4d7c-9285-cefad05006a5" containerName="revoke-keys" Nov 01 23:11:01.700626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.700599 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65153a3e-b028-4d7c-9285-cefad05006a5" containerName="revoke-keys" Nov 01 23:11:01.700740 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.700730 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d5338e-6670-4c89-adbe-bb8850cbaaa4" containerName="istio-proxy" Nov 01 23:11:01.700778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.700741 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="65153a3e-b028-4d7c-9285-cefad05006a5" containerName="revoke-keys" Nov 01 23:11:01.705217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.705198 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.709066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.709045 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:11:01.710041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.710016 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:11:01.712482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.712460 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk"] Nov 01 23:11:01.715083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.715060 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:11:01.715216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.715100 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:11:01.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.723200 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.723340 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xwl\" (UniqueName: \"kubernetes.io/projected/39948769-7fef-4a5c-aedd-b67ec3528417-kube-api-access-v6xwl\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.737603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.723439 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.824734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.824694 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.824912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.824789 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xwl\" (UniqueName: \"kubernetes.io/projected/39948769-7fef-4a5c-aedd-b67ec3528417-kube-api-access-v6xwl\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.824912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.824863 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.825460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.825441 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.827362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.827336 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:01.833302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:01.833270 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xwl\" (UniqueName: \"kubernetes.io/projected/39948769-7fef-4a5c-aedd-b67ec3528417-kube-api-access-v6xwl\") pod \"maas-api-revoke-sub-a595d015-x74qk\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:02.022549 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.022463 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:02.064208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.064169 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:02.305424 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:02.305383 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39948769_7fef_4a5c_aedd_b67ec3528417.slice/crio-917ba97c23d9390aa5f019d5a5631be5885ea088bf7415ee22fcc3a93978cdd1 WatchSource:0}: Error finding container 917ba97c23d9390aa5f019d5a5631be5885ea088bf7415ee22fcc3a93978cdd1: Status 404 returned error can't find the container with id 917ba97c23d9390aa5f019d5a5631be5885ea088bf7415ee22fcc3a93978cdd1 Nov 01 23:11:02.305724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.305699 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk"] Nov 01 23:11:02.448137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.448102 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" event={"ID":"39948769-7fef-4a5c-aedd-b67ec3528417","Type":"ContainerStarted","Data":"917ba97c23d9390aa5f019d5a5631be5885ea088bf7415ee22fcc3a93978cdd1"} Nov 01 23:11:02.706303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.706271 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:02.742732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.742669 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts"] Nov 01 23:11:02.742971 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.742903 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" podUID="c7697c68-e0f4-4913-9d4c-1294ad63a74a" containerName="payload-pre-processing" containerID="cri-o://a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780" gracePeriod=30 Nov 01 23:11:02.766478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.766443 4981 scope.go:122] "RemoveContainer" containerID="c6c0e93528724c5b17af7633fd449477e9e88d88f2bfd21890d3e1dd1ae13eff" Nov 01 23:11:02.806836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.806806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:02.860342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.860310 4981 scope.go:122] "RemoveContainer" containerID="3b55d023f88d29a07c5dd995f9425b252396c845cf5f21c2e120af23fdc0a405" Nov 01 23:11:02.912913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:02.912884 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:03.052725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.048307 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb"] Nov 01 23:11:03.052725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.048372 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-46ce02-92kmb"] Nov 01 23:11:03.089107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.086900 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:03.183488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.183452 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:11:03.218857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.218816 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:03.270610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.270526 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7697c68-e0f4-4913-9d4c-1294ad63a74a-kube-api-access-9sdd6\" (UniqueName: \"kubernetes.io/projected/c7697c68-e0f4-4913-9d4c-1294ad63a74a-kube-api-access-9sdd6\") pod \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " Nov 01 23:11:03.271103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.271072 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7697c68-e0f4-4913-9d4c-1294ad63a74a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7697c68-e0f4-4913-9d4c-1294ad63a74a-plugins-config-volume\") pod \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\" (UID: \"c7697c68-e0f4-4913-9d4c-1294ad63a74a\") " Nov 01 23:11:03.272335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.272304 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7697c68-e0f4-4913-9d4c-1294ad63a74a-plugins-config-volume" pod "c7697c68-e0f4-4913-9d4c-1294ad63a74a" (UID: "c7697c68-e0f4-4913-9d4c-1294ad63a74a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:03.275648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.275619 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7697c68-e0f4-4913-9d4c-1294ad63a74a-kube-api-access-9sdd6" pod "c7697c68-e0f4-4913-9d4c-1294ad63a74a" (UID: "c7697c68-e0f4-4913-9d4c-1294ad63a74a"). InnerVolumeSpecName "kube-api-access-9sdd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:03.373596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.373523 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sdd6\" (UniqueName: \"kubernetes.io/projected/c7697c68-e0f4-4913-9d4c-1294ad63a74a-kube-api-access-9sdd6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.373596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.373569 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7697c68-e0f4-4913-9d4c-1294ad63a74a-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:03.458100 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.457997 4981 generic.go:356] "Generic (PLEG): container finished" podID="39948769-7fef-4a5c-aedd-b67ec3528417" containerID="899bdd9fa93b995912c922333eedd9b75fc34a26d848165fef7a4156342db50b" exitCode=0 Nov 01 23:11:03.458100 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.458072 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" event={"ID":"39948769-7fef-4a5c-aedd-b67ec3528417","Type":"ContainerDied","Data":"899bdd9fa93b995912c922333eedd9b75fc34a26d848165fef7a4156342db50b"} Nov 01 23:11:03.458346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.458153 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:03.459621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.459567 4981 generic.go:356] "Generic (PLEG): container finished" podID="c7697c68-e0f4-4913-9d4c-1294ad63a74a" containerID="a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780" exitCode=0 Nov 01 23:11:03.459747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.459635 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" Nov 01 23:11:03.459747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.459660 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" event={"ID":"c7697c68-e0f4-4913-9d4c-1294ad63a74a","Type":"ContainerDied","Data":"a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780"} Nov 01 23:11:03.459747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.459707 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts" event={"ID":"c7697c68-e0f4-4913-9d4c-1294ad63a74a","Type":"ContainerDied","Data":"186c665a2d709a84eb8e9a616a978147af78ef2cb26675bd3ff073fc7360b71e"} Nov 01 23:11:03.459747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.459734 4981 scope.go:122] "RemoveContainer" containerID="a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780" Nov 01 23:11:03.471455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.471428 4981 scope.go:122] "RemoveContainer" containerID="a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780" Nov 01 23:11:03.471808 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:03.471787 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780\": container with ID starting with a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780 not found: ID does not exist" containerID="a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780" Nov 01 23:11:03.471906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.471816 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780"} err="failed to get container status \"a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780\": rpc error: code = NotFound desc = could not find container \"a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780\": container with ID starting with a525120d4bc2a5b96cf3957bc356874d2bc65274f538dfebd4afb01dffda9780 not found: ID does not exist" Nov 01 23:11:03.503911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.503872 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts"] Nov 01 23:11:03.510560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:03.510524 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-46ce02-5bd899c6d7-q72ts"] Nov 01 23:11:04.259475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.259444 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:04.632471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.632446 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:04.690502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.690467 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-xks-ca\") pod \"39948769-7fef-4a5c-aedd-b67ec3528417\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " Nov 01 23:11:04.690705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.690665 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-openshift-ca\") pod \"39948769-7fef-4a5c-aedd-b67ec3528417\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " Nov 01 23:11:04.690829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.690809 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39948769-7fef-4a5c-aedd-b67ec3528417-kube-api-access-v6xwl\" (UniqueName: \"kubernetes.io/projected/39948769-7fef-4a5c-aedd-b67ec3528417-kube-api-access-v6xwl\") pod \"39948769-7fef-4a5c-aedd-b67ec3528417\" (UID: \"39948769-7fef-4a5c-aedd-b67ec3528417\") " Nov 01 23:11:04.691103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.691075 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-openshift-ca" pod "39948769-7fef-4a5c-aedd-b67ec3528417" (UID: "39948769-7fef-4a5c-aedd-b67ec3528417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:04.691303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.691284 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:04.692961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.692939 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39948769-7fef-4a5c-aedd-b67ec3528417-kube-api-access-v6xwl" pod "39948769-7fef-4a5c-aedd-b67ec3528417" (UID: "39948769-7fef-4a5c-aedd-b67ec3528417"). InnerVolumeSpecName "kube-api-access-v6xwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:04.693253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.693229 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-xks-ca" pod "39948769-7fef-4a5c-aedd-b67ec3528417" (UID: "39948769-7fef-4a5c-aedd-b67ec3528417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:04.736236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.736198 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541276ef-bf48-47fb-b6ac-67d81069b23e" path="/var/lib/kubelet/pods/541276ef-bf48-47fb-b6ac-67d81069b23e/volumes" Nov 01 23:11:04.736800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.736783 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7697c68-e0f4-4913-9d4c-1294ad63a74a" path="/var/lib/kubelet/pods/c7697c68-e0f4-4913-9d4c-1294ad63a74a/volumes" Nov 01 23:11:04.793078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.792987 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39948769-7fef-4a5c-aedd-b67ec3528417-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:04.793078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.793023 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xwl\" (UniqueName: \"kubernetes.io/projected/39948769-7fef-4a5c-aedd-b67ec3528417-kube-api-access-v6xwl\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:04.968693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:04.968663 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:05.286808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:05.286773 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc"] Nov 01 23:11:05.287263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:05.286962 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" containerID="cri-o://095e97bce8e545776f55c677c6f8f31f08f48a1e11ff8b88fb3c889c40cc3b45" gracePeriod=30 Nov 01 23:11:05.479282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:05.479252 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" Nov 01 23:11:05.479460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:05.479277 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk" event={"ID":"39948769-7fef-4a5c-aedd-b67ec3528417","Type":"ContainerDied","Data":"917ba97c23d9390aa5f019d5a5631be5885ea088bf7415ee22fcc3a93978cdd1"} Nov 01 23:11:05.479460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:05.479325 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917ba97c23d9390aa5f019d5a5631be5885ea088bf7415ee22fcc3a93978cdd1" Nov 01 23:11:05.795176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:05.795143 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:06.301813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:06.301784 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:06.404779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:06.404746 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:06.520559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:06.520462 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:06.580806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:06.580715 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z"] Nov 01 23:11:06.580993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:06.580969 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" containerID="cri-o://26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4" gracePeriod=30 Nov 01 23:11:06.622655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:06.622608 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:07.029891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.029854 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6"] Nov 01 23:11:07.030618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.030561 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c7697c68-e0f4-4913-9d4c-1294ad63a74a" containerName="payload-pre-processing" Nov 01 23:11:07.030618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.030599 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7697c68-e0f4-4913-9d4c-1294ad63a74a" containerName="payload-pre-processing" Nov 01 23:11:07.030831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.030637 4981 cpu_manager.go:414] "containerMap: removing container" podUID="39948769-7fef-4a5c-aedd-b67ec3528417" containerName="revoke-keys" Nov 01 23:11:07.030831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.030645 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39948769-7fef-4a5c-aedd-b67ec3528417" containerName="revoke-keys" Nov 01 23:11:07.030831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.030788 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7697c68-e0f4-4913-9d4c-1294ad63a74a" containerName="payload-pre-processing" Nov 01 23:11:07.030831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.030807 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="39948769-7fef-4a5c-aedd-b67ec3528417" containerName="revoke-keys" Nov 01 23:11:07.035976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.035955 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.039724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.039701 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:11:07.040416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.040398 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:11:07.042359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.042331 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6"] Nov 01 23:11:07.130303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.130272 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:11:07.130486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.130445 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:11:07.137144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.137114 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.137245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.137227 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.137307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.137293 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8xf\" (UniqueName: \"kubernetes.io/projected/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-kube-api-access-pd8xf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.238223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.238191 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.238384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.238288 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.238384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.238338 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8xf\" (UniqueName: \"kubernetes.io/projected/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-kube-api-access-pd8xf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.238884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.238856 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.240746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.240728 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.247188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.247168 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8xf\" (UniqueName: \"kubernetes.io/projected/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-kube-api-access-pd8xf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.340523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.340405 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:07.355360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.355336 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:07.655303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:07.655266 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6"] Nov 01 23:11:07.655500 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:07.655478 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a972026_e3ba_4f70_b6f8_6b37d276a4a9.slice/crio-727f8a6d41785ccad94892d7288e50ec588d557694d8eb62ecc7f2f5c5fa714c WatchSource:0}: Error finding container 727f8a6d41785ccad94892d7288e50ec588d557694d8eb62ecc7f2f5c5fa714c: Status 404 returned error can't find the container with id 727f8a6d41785ccad94892d7288e50ec588d557694d8eb62ecc7f2f5c5fa714c Nov 01 23:11:08.056341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:08.056307 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:08.506929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:08.506899 4981 generic.go:356] "Generic (PLEG): container finished" podID="0a972026-e3ba-4f70-b6f8-6b37d276a4a9" containerID="29a719cc8ab90633bdb35674da03095752b498c436d3d287a1e467199e3ddd21" exitCode=0 Nov 01 23:11:08.507240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:08.506960 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" event={"ID":"0a972026-e3ba-4f70-b6f8-6b37d276a4a9","Type":"ContainerDied","Data":"29a719cc8ab90633bdb35674da03095752b498c436d3d287a1e467199e3ddd21"} Nov 01 23:11:08.507240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:08.507001 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" event={"ID":"0a972026-e3ba-4f70-b6f8-6b37d276a4a9","Type":"ContainerStarted","Data":"727f8a6d41785ccad94892d7288e50ec588d557694d8eb62ecc7f2f5c5fa714c"} Nov 01 23:11:08.669443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:08.669410 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:09.453735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.453699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:09.687617 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.687590 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:09.759074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.758992 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:09.774862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.774838 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-openshift-ca\") pod \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " Nov 01 23:11:09.774995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.774885 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-kube-api-access-pd8xf\" (UniqueName: \"kubernetes.io/projected/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-kube-api-access-pd8xf\") pod \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " Nov 01 23:11:09.774995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.774934 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-xks-ca\") pod \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\" (UID: \"0a972026-e3ba-4f70-b6f8-6b37d276a4a9\") " Nov 01 23:11:09.775257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.775228 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-openshift-ca" pod "0a972026-e3ba-4f70-b6f8-6b37d276a4a9" (UID: "0a972026-e3ba-4f70-b6f8-6b37d276a4a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:09.775640 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.775621 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:09.777046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.777017 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-xks-ca" pod "0a972026-e3ba-4f70-b6f8-6b37d276a4a9" (UID: "0a972026-e3ba-4f70-b6f8-6b37d276a4a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:09.777137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.777076 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-kube-api-access-pd8xf" pod "0a972026-e3ba-4f70-b6f8-6b37d276a4a9" (UID: "0a972026-e3ba-4f70-b6f8-6b37d276a4a9"). InnerVolumeSpecName "kube-api-access-pd8xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:09.842910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.842872 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kxkqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:11:09.843091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.842933 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:11:09.860088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.860054 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:09.876868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.876846 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd8xf\" (UniqueName: \"kubernetes.io/projected/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-kube-api-access-pd8xf\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:09.876868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:09.876869 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a972026-e3ba-4f70-b6f8-6b37d276a4a9-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.078746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.078660 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:10.177186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.177143 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:10.488845 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.488805 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v"] Nov 01 23:11:10.489659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.489630 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0a972026-e3ba-4f70-b6f8-6b37d276a4a9" containerName="revoke-keys" Nov 01 23:11:10.489659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.489650 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a972026-e3ba-4f70-b6f8-6b37d276a4a9" containerName="revoke-keys" Nov 01 23:11:10.489856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.489835 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a972026-e3ba-4f70-b6f8-6b37d276a4a9" containerName="revoke-keys" Nov 01 23:11:10.493635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.493608 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.498260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.498231 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v"] Nov 01 23:11:10.540292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.540265 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" Nov 01 23:11:10.540442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.540262 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6" event={"ID":"0a972026-e3ba-4f70-b6f8-6b37d276a4a9","Type":"ContainerDied","Data":"727f8a6d41785ccad94892d7288e50ec588d557694d8eb62ecc7f2f5c5fa714c"} Nov 01 23:11:10.540508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.540445 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727f8a6d41785ccad94892d7288e50ec588d557694d8eb62ecc7f2f5c5fa714c" Nov 01 23:11:10.542398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.542378 4981 generic.go:356] "Generic (PLEG): container finished" podID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerID="095e97bce8e545776f55c677c6f8f31f08f48a1e11ff8b88fb3c889c40cc3b45" exitCode=0 Nov 01 23:11:10.542506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.542449 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" event={"ID":"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c","Type":"ContainerDied","Data":"095e97bce8e545776f55c677c6f8f31f08f48a1e11ff8b88fb3c889c40cc3b45"} Nov 01 23:11:10.548716 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.548697 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 23:11:10.587544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587516 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-data\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.587760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587637 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istiod-ca-cert\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.587760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587693 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-certs\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.587760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587724 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-token\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-token\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.587948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587789 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-socket\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.587948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587820 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-kube-api-access-gs5fc\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-kube-api-access-gs5fc\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.587948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587862 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-podinfo\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.587948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587864 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-data" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:10.587948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587889 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-envoy\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.588274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.587979 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-certs" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:10.588274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588045 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-credential-socket\") pod \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\" (UID: \"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c\") " Nov 01 23:11:10.588274 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588160 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-socket" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:10.588459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588423 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbvl\" (UniqueName: \"kubernetes.io/projected/0bb20711-fb58-4851-9272-3f4ee464c200-kube-api-access-9xbvl\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.588536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588511 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-credential-socket" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:10.588658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588635 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.588739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588665 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istiod-ca-cert" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:10.588817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588789 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.589094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.588999 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.589094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.589025 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.589094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.589041 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.589094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.589056 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.589094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.589071 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.590734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.590710 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-envoy" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:10.590930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.590911 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-podinfo" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:10.591062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.591042 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-kube-api-access-gs5fc" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "kube-api-access-gs5fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:10.591215 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.591193 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-token" pod "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" (UID: "a34fccec-9a46-4bc1-87ed-5de63a8b0a1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:10.690220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.690185 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbvl\" (UniqueName: \"kubernetes.io/projected/0bb20711-fb58-4851-9272-3f4ee464c200-kube-api-access-9xbvl\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.690762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.690278 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.690762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.690337 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.690762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.690469 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5fc\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-kube-api-access-gs5fc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.690762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.690481 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.690762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.690491 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.690762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.690500 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:10.691042 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.691021 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.692809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.692782 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.699113 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.699090 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbvl\" (UniqueName: \"kubernetes.io/projected/0bb20711-fb58-4851-9272-3f4ee464c200-kube-api-access-9xbvl\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.815026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.814995 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:10.961436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:10.961407 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:11.098264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.098237 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v"] Nov 01 23:11:11.098548 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:11.098522 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb20711_fb58_4851_9272_3f4ee464c200.slice/crio-6fda165c07cb1f932f6c116029215a47c69a2c9b6d4d6d0649fb6cdf416cfc77 WatchSource:0}: Error finding container 6fda165c07cb1f932f6c116029215a47c69a2c9b6d4d6d0649fb6cdf416cfc77: Status 404 returned error can't find the container with id 6fda165c07cb1f932f6c116029215a47c69a2c9b6d4d6d0649fb6cdf416cfc77 Nov 01 23:11:11.553216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.553124 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" event={"ID":"0bb20711-fb58-4851-9272-3f4ee464c200","Type":"ContainerStarted","Data":"f88fcbe9249d34a82e4b458b3a4c79da1eae9a465f85d55c3493195dea155b72"} Nov 01 23:11:11.553216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.553186 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" event={"ID":"0bb20711-fb58-4851-9272-3f4ee464c200","Type":"ContainerStarted","Data":"6fda165c07cb1f932f6c116029215a47c69a2c9b6d4d6d0649fb6cdf416cfc77"} Nov 01 23:11:11.554504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.554483 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" Nov 01 23:11:11.554655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.554526 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc" event={"ID":"a34fccec-9a46-4bc1-87ed-5de63a8b0a1c","Type":"ContainerDied","Data":"e4669a6b9702150f4eb73ee7e05c4d25bd4557e13fd8afdaad114574556b37ec"} Nov 01 23:11:11.554655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.554595 4981 scope.go:122] "RemoveContainer" containerID="095e97bce8e545776f55c677c6f8f31f08f48a1e11ff8b88fb3c889c40cc3b45" Nov 01 23:11:11.573957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.573913 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" podStartSLOduration=1.573900895 podStartE2EDuration="1.573900895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:11.572046138 +0000 UTC m=+2836.999924071" watchObservedRunningTime="2026-11-01 23:11:11.573900895 +0000 UTC m=+2837.001778826" Nov 01 23:11:11.590533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.590506 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc"] Nov 01 23:11:11.599200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.599179 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kxkqc"] Nov 01 23:11:11.824554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.824470 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:11.889063 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:11.889041 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 23:11:12.007442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.007410 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-socket\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.007624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.007488 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-podinfo\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.007624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.007538 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-kube-api-access-s6dvk\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-kube-api-access-s6dvk\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.007781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.007637 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c078984-19f8-4130-9ec5-6ae240e8557d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c078984-19f8-4130-9ec5-6ae240e8557d-istiod-ca-cert\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.007781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.007718 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-envoy\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.007781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.007739 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-socket" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:12.007781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.007774 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-certs\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.008045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008018 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-certs" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:12.008113 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008060 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c078984-19f8-4130-9ec5-6ae240e8557d-istiod-ca-cert" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:12.008191 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008165 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-data\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.008358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008326 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-credential-socket\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.008446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008420 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-token\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-token\") pod \"6c078984-19f8-4130-9ec5-6ae240e8557d\" (UID: \"6c078984-19f8-4130-9ec5-6ae240e8557d\") " Nov 01 23:11:12.008507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008491 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-data" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:12.008882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008677 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-credential-socket" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:12.009014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.008994 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.009084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.009021 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.009084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.009037 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c078984-19f8-4130-9ec5-6ae240e8557d-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.009084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.009050 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.009084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.009063 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.010149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.010123 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-envoy" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:12.010265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.010189 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-podinfo" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:12.010336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.010313 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-kube-api-access-s6dvk" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "kube-api-access-s6dvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:12.010997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.010967 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-token" pod "6c078984-19f8-4130-9ec5-6ae240e8557d" (UID: "6c078984-19f8-4130-9ec5-6ae240e8557d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:12.110189 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.110107 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6dvk\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-kube-api-access-s6dvk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.110189 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.110140 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.110189 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.110151 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.110189 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.110160 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c078984-19f8-4130-9ec5-6ae240e8557d-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:12.429679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.428506 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:12.676608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.674477 4981 generic.go:356] "Generic (PLEG): container finished" podID="0bb20711-fb58-4851-9272-3f4ee464c200" containerID="f88fcbe9249d34a82e4b458b3a4c79da1eae9a465f85d55c3493195dea155b72" exitCode=0 Nov 01 23:11:12.676608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.674607 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" event={"ID":"0bb20711-fb58-4851-9272-3f4ee464c200","Type":"ContainerDied","Data":"f88fcbe9249d34a82e4b458b3a4c79da1eae9a465f85d55c3493195dea155b72"} Nov 01 23:11:12.714650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.714199 4981 generic.go:356] "Generic (PLEG): container finished" podID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerID="26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4" exitCode=0 Nov 01 23:11:12.714650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.714271 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" event={"ID":"6c078984-19f8-4130-9ec5-6ae240e8557d","Type":"ContainerDied","Data":"26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4"} Nov 01 23:11:12.714650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.714356 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" Nov 01 23:11:12.737623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.716005 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z" event={"ID":"6c078984-19f8-4130-9ec5-6ae240e8557d","Type":"ContainerDied","Data":"f21eb61636b5ec0b708760296902e7a6be878b8ed4dcab0928a83b07d3784826"} Nov 01 23:11:12.737623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.716069 4981 scope.go:122] "RemoveContainer" containerID="26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4" Nov 01 23:11:12.805058 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.802317 4981 scope.go:122] "RemoveContainer" containerID="386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18" Nov 01 23:11:12.846172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.846139 4981 scope.go:122] "RemoveContainer" containerID="26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4" Nov 01 23:11:12.847524 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:12.847496 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4\": container with ID starting with 26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4 not found: ID does not exist" containerID="26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4" Nov 01 23:11:12.847722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.847689 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4"} err="failed to get container status \"26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4\": rpc error: code = NotFound desc = could not find container \"26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4\": container with ID starting with 26f014384e286c5d2aa844027c81a3bd065a573b8717ca42c3cd0c8f1b54e9a4 not found: ID does not exist" Nov 01 23:11:12.847852 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.847837 4981 scope.go:122] "RemoveContainer" containerID="386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18" Nov 01 23:11:12.848328 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:12.848306 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18\": container with ID starting with 386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18 not found: ID does not exist" containerID="386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18" Nov 01 23:11:12.848494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.848467 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18"} err="failed to get container status \"386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18\": rpc error: code = NotFound desc = could not find container \"386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18\": container with ID starting with 386f3c6aeaab80d3d34ff5ccc6d26ab1e08cf8107ddbcf6f446d95c9e8b0ab18 not found: ID does not exist" Nov 01 23:11:12.850626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.850415 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" path="/var/lib/kubelet/pods/a34fccec-9a46-4bc1-87ed-5de63a8b0a1c/volumes" Nov 01 23:11:12.863707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.863653 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z"] Nov 01 23:11:12.868258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.864290 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c6cec6-openshift-default-6c9845ccf7-cmq9z"] Nov 01 23:11:12.951634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:12.949609 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:13.877431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:13.877408 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:13.900491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:13.900456 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:14.007379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.007299 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-openshift-ca\") pod \"0bb20711-fb58-4851-9272-3f4ee464c200\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " Nov 01 23:11:14.007379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.007356 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-xks-ca\") pod \"0bb20711-fb58-4851-9272-3f4ee464c200\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " Nov 01 23:11:14.007614 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.007497 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb20711-fb58-4851-9272-3f4ee464c200-kube-api-access-9xbvl\" (UniqueName: \"kubernetes.io/projected/0bb20711-fb58-4851-9272-3f4ee464c200-kube-api-access-9xbvl\") pod \"0bb20711-fb58-4851-9272-3f4ee464c200\" (UID: \"0bb20711-fb58-4851-9272-3f4ee464c200\") " Nov 01 23:11:14.007695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.007673 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-openshift-ca" pod "0bb20711-fb58-4851-9272-3f4ee464c200" (UID: "0bb20711-fb58-4851-9272-3f4ee464c200"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:14.007948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.007931 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:14.009585 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.009545 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-xks-ca" pod "0bb20711-fb58-4851-9272-3f4ee464c200" (UID: "0bb20711-fb58-4851-9272-3f4ee464c200"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:14.009686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.009601 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb20711-fb58-4851-9272-3f4ee464c200-kube-api-access-9xbvl" pod "0bb20711-fb58-4851-9272-3f4ee464c200" (UID: "0bb20711-fb58-4851-9272-3f4ee464c200"). InnerVolumeSpecName "kube-api-access-9xbvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:14.109289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.109260 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbvl\" (UniqueName: \"kubernetes.io/projected/0bb20711-fb58-4851-9272-3f4ee464c200-kube-api-access-9xbvl\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:14.109289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.109288 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb20711-fb58-4851-9272-3f4ee464c200-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:14.527726 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:14.527688 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:11:14.547728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.547700 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:14.734319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.734292 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" Nov 01 23:11:14.737496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.737469 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" path="/var/lib/kubelet/pods/6c078984-19f8-4130-9ec5-6ae240e8557d/volumes" Nov 01 23:11:14.738146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.738125 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v" event={"ID":"0bb20711-fb58-4851-9272-3f4ee464c200","Type":"ContainerDied","Data":"6fda165c07cb1f932f6c116029215a47c69a2c9b6d4d6d0649fb6cdf416cfc77"} Nov 01 23:11:14.738244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:14.738172 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fda165c07cb1f932f6c116029215a47c69a2c9b6d4d6d0649fb6cdf416cfc77" Nov 01 23:11:15.247674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:15.247640 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:15.531851 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:15.531773 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:15.917500 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:15.917424 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:16.173327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:16.173289 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:16.665104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:16.665028 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:17.110187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.110155 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:17.180603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.178806 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm"] Nov 01 23:11:17.185439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185405 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" Nov 01 23:11:17.185439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185438 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185473 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0bb20711-fb58-4851-9272-3f4ee464c200" containerName="revoke-keys" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185483 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb20711-fb58-4851-9272-3f4ee464c200" containerName="revoke-keys" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185519 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185526 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185793 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185832 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb20711-fb58-4851-9272-3f4ee464c200" containerName="revoke-keys" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185848 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34fccec-9a46-4bc1-87ed-5de63a8b0a1c" containerName="istio-proxy" Nov 01 23:11:17.186341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.185856 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" Nov 01 23:11:17.195610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.191592 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm"] Nov 01 23:11:17.195610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.191739 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.197216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.197195 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:11:17.198923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.198899 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:11:17.199070 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.198989 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:11:17.199070 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.199036 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:11:17.343233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.343203 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:17.362671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.362263 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.362671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.362380 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.362671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.362479 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9f7\" (UniqueName: \"kubernetes.io/projected/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-kube-api-access-ql9f7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.442437 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.442408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:17.463896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.463866 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9f7\" (UniqueName: \"kubernetes.io/projected/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-kube-api-access-ql9f7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.464054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.464004 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.464102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.464053 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.464703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.464677 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.466458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.466436 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.471910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.471886 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9f7\" (UniqueName: \"kubernetes.io/projected/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-kube-api-access-ql9f7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.511092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.511066 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:17.850377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.850341 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:17.915601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:17.914832 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm"] Nov 01 23:11:17.942918 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:17.942563 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecad335_0e3c_4911_bb34_3e8ab8d516fc.slice/crio-ca7853af1b789dfeacc4107dd7a46e558bc5eed1ecbee4835ebf77a7b8afc658 WatchSource:0}: Error finding container ca7853af1b789dfeacc4107dd7a46e558bc5eed1ecbee4835ebf77a7b8afc658: Status 404 returned error can't find the container with id ca7853af1b789dfeacc4107dd7a46e558bc5eed1ecbee4835ebf77a7b8afc658 Nov 01 23:11:18.775968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:18.775931 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" event={"ID":"1ecad335-0e3c-4911-bb34-3e8ab8d516fc","Type":"ContainerStarted","Data":"24e28925ecfe5fbb340b749a21ded7a33bfa38ef13ef2abc895add8c30f03476"} Nov 01 23:11:18.776153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:18.775995 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" event={"ID":"1ecad335-0e3c-4911-bb34-3e8ab8d516fc","Type":"ContainerStarted","Data":"ca7853af1b789dfeacc4107dd7a46e558bc5eed1ecbee4835ebf77a7b8afc658"} Nov 01 23:11:18.824709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:18.824467 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:18.827773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:18.827705 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" podStartSLOduration=1.8276860099999999 podStartE2EDuration="1.82768601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:18.794052482 +0000 UTC m=+2844.221930412" watchObservedRunningTime="2026-11-01 23:11:18.82768601 +0000 UTC m=+2844.255563943" Nov 01 23:11:19.033941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.033868 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7"] Nov 01 23:11:19.034495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.034481 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" Nov 01 23:11:19.034541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.034496 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c078984-19f8-4130-9ec5-6ae240e8557d" containerName="istio-proxy" Nov 01 23:11:19.038195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.038178 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.048811 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.047296 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7"] Nov 01 23:11:19.188878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.188841 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dgv\" (UniqueName: \"kubernetes.io/projected/f8c4903d-3833-40e5-88ee-d0369d2d4092-kube-api-access-x5dgv\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.189062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.188942 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.189062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.189030 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.290043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.289954 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dgv\" (UniqueName: \"kubernetes.io/projected/f8c4903d-3833-40e5-88ee-d0369d2d4092-kube-api-access-x5dgv\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.290043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.290013 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.290294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.290052 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.290768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.290747 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.292494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.292466 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.298931 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.298907 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dgv\" (UniqueName: \"kubernetes.io/projected/f8c4903d-3833-40e5-88ee-d0369d2d4092-kube-api-access-x5dgv\") pod \"maas-api-revoke-sub-5b43b1ab-m5kv7\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.356663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.356633 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:19.469940 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.469891 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:19.641567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.641535 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7"] Nov 01 23:11:19.642343 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:19.642314 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c4903d_3833_40e5_88ee_d0369d2d4092.slice/crio-9f8be8a81acee5ecfdea880c4cb2332eead2d6f273507fabf0eb362d3a9dd93f WatchSource:0}: Error finding container 9f8be8a81acee5ecfdea880c4cb2332eead2d6f273507fabf0eb362d3a9dd93f: Status 404 returned error can't find the container with id 9f8be8a81acee5ecfdea880c4cb2332eead2d6f273507fabf0eb362d3a9dd93f Nov 01 23:11:19.786727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.786695 4981 generic.go:356] "Generic (PLEG): container finished" podID="1ecad335-0e3c-4911-bb34-3e8ab8d516fc" containerID="24e28925ecfe5fbb340b749a21ded7a33bfa38ef13ef2abc895add8c30f03476" exitCode=0 Nov 01 23:11:19.786919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.786828 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" event={"ID":"1ecad335-0e3c-4911-bb34-3e8ab8d516fc","Type":"ContainerDied","Data":"24e28925ecfe5fbb340b749a21ded7a33bfa38ef13ef2abc895add8c30f03476"} Nov 01 23:11:19.788167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:19.788145 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" event={"ID":"f8c4903d-3833-40e5-88ee-d0369d2d4092","Type":"ContainerStarted","Data":"9f8be8a81acee5ecfdea880c4cb2332eead2d6f273507fabf0eb362d3a9dd93f"} Nov 01 23:11:20.160930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:20.160852 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:20.611817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:20.611780 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn"] Nov 01 23:11:20.658431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:20.658388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:20.774719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:20.774647 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:20.797122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:20.797092 4981 generic.go:356] "Generic (PLEG): container finished" podID="f8c4903d-3833-40e5-88ee-d0369d2d4092" containerID="618f86c9c82999794bdd7ffc7d8c43f258fe86c687172e4ed6665c043b7a929b" exitCode=0 Nov 01 23:11:20.797294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:20.797174 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" event={"ID":"f8c4903d-3833-40e5-88ee-d0369d2d4092","Type":"ContainerDied","Data":"618f86c9c82999794bdd7ffc7d8c43f258fe86c687172e4ed6665c043b7a929b"} Nov 01 23:11:20.858750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:20.858718 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:21.029537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.029469 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:21.126135 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.126102 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:21.126309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.126115 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-xks-ca\") pod \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " Nov 01 23:11:21.126309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.126283 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-openshift-ca\") pod \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " Nov 01 23:11:21.126397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.126380 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-kube-api-access-ql9f7\" (UniqueName: \"kubernetes.io/projected/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-kube-api-access-ql9f7\") pod \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\" (UID: \"1ecad335-0e3c-4911-bb34-3e8ab8d516fc\") " Nov 01 23:11:21.126723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.126697 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-openshift-ca" pod "1ecad335-0e3c-4911-bb34-3e8ab8d516fc" (UID: "1ecad335-0e3c-4911-bb34-3e8ab8d516fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:21.126988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.126970 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:21.128234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.128210 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-xks-ca" pod "1ecad335-0e3c-4911-bb34-3e8ab8d516fc" (UID: "1ecad335-0e3c-4911-bb34-3e8ab8d516fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:21.128472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.128453 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-kube-api-access-ql9f7" pod "1ecad335-0e3c-4911-bb34-3e8ab8d516fc" (UID: "1ecad335-0e3c-4911-bb34-3e8ab8d516fc"). InnerVolumeSpecName "kube-api-access-ql9f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:21.228147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.228111 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:21.228147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.228148 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9f7\" (UniqueName: \"kubernetes.io/projected/1ecad335-0e3c-4911-bb34-3e8ab8d516fc-kube-api-access-ql9f7\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:21.692345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.692304 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:21.806615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.806567 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" Nov 01 23:11:21.806615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.806597 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm" event={"ID":"1ecad335-0e3c-4911-bb34-3e8ab8d516fc","Type":"ContainerDied","Data":"ca7853af1b789dfeacc4107dd7a46e558bc5eed1ecbee4835ebf77a7b8afc658"} Nov 01 23:11:21.806865 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.806658 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca7853af1b789dfeacc4107dd7a46e558bc5eed1ecbee4835ebf77a7b8afc658" Nov 01 23:11:21.837348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.837320 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:21.889155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.889106 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6"] Nov 01 23:11:21.893068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.893036 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-947846-ptfb6"] Nov 01 23:11:21.921605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:21.921560 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:22.049550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.049524 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:22.121196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.121150 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:22.147920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.147884 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-openshift-ca\") pod \"f8c4903d-3833-40e5-88ee-d0369d2d4092\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " Nov 01 23:11:22.148083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.147944 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c4903d-3833-40e5-88ee-d0369d2d4092-kube-api-access-x5dgv\" (UniqueName: \"kubernetes.io/projected/f8c4903d-3833-40e5-88ee-d0369d2d4092-kube-api-access-x5dgv\") pod \"f8c4903d-3833-40e5-88ee-d0369d2d4092\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " Nov 01 23:11:22.148083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.147979 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-xks-ca\") pod \"f8c4903d-3833-40e5-88ee-d0369d2d4092\" (UID: \"f8c4903d-3833-40e5-88ee-d0369d2d4092\") " Nov 01 23:11:22.148307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.148273 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-openshift-ca" pod "f8c4903d-3833-40e5-88ee-d0369d2d4092" (UID: "f8c4903d-3833-40e5-88ee-d0369d2d4092"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:22.148762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.148738 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:22.150307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.150217 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-xks-ca" pod "f8c4903d-3833-40e5-88ee-d0369d2d4092" (UID: "f8c4903d-3833-40e5-88ee-d0369d2d4092"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:22.150484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.150454 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c4903d-3833-40e5-88ee-d0369d2d4092-kube-api-access-x5dgv" pod "f8c4903d-3833-40e5-88ee-d0369d2d4092" (UID: "f8c4903d-3833-40e5-88ee-d0369d2d4092"). InnerVolumeSpecName "kube-api-access-x5dgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:22.249928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.249898 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5dgv\" (UniqueName: \"kubernetes.io/projected/f8c4903d-3833-40e5-88ee-d0369d2d4092-kube-api-access-x5dgv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:22.249928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.249929 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c4903d-3833-40e5-88ee-d0369d2d4092-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:22.336986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.336885 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:22.735008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.734976 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a972026-e3ba-4f70-b6f8-6b37d276a4a9" path="/var/lib/kubelet/pods/0a972026-e3ba-4f70-b6f8-6b37d276a4a9/volumes" Nov 01 23:11:22.815372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.815345 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" Nov 01 23:11:22.815541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.815391 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7" event={"ID":"f8c4903d-3833-40e5-88ee-d0369d2d4092","Type":"ContainerDied","Data":"9f8be8a81acee5ecfdea880c4cb2332eead2d6f273507fabf0eb362d3a9dd93f"} Nov 01 23:11:22.815541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.815435 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8be8a81acee5ecfdea880c4cb2332eead2d6f273507fabf0eb362d3a9dd93f" Nov 01 23:11:22.999462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:22.999372 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:23.846534 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:23.846500 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:24.566751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:24.566718 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:25.271730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:25.271691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:25.613311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:25.613229 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" podUID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" containerName="payload-pre-processing" containerID="cri-o://f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac" gracePeriod=25 Nov 01 23:11:25.969702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:25.969656 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:26.655301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:26.655254 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:27.256149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:27.256115 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh"] Nov 01 23:11:27.271292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:27.271263 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp"] Nov 01 23:11:27.405523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:27.405493 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:27.549426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:27.549132 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:27.875116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:27.875011 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:28.034759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.034726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:28.049836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.049795 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm"] Nov 01 23:11:28.057046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.057012 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-7366bf-x6fcm"] Nov 01 23:11:28.137788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.137657 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:28.300495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.300457 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:28.341717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.341689 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v"] Nov 01 23:11:28.345930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.345905 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a9f94e-ckd9v"] Nov 01 23:11:28.401837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.401757 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:28.521039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.520754 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:28.735461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.735428 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb20711-fb58-4851-9272-3f4ee464c200" path="/var/lib/kubelet/pods/0bb20711-fb58-4851-9272-3f4ee464c200/volumes" Nov 01 23:11:28.735950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.735935 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecad335-0e3c-4911-bb34-3e8ab8d516fc" path="/var/lib/kubelet/pods/1ecad335-0e3c-4911-bb34-3e8ab8d516fc/volumes" Nov 01 23:11:28.795050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:28.795021 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:29.229613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.229571 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:29.342634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.342601 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l"] Nov 01 23:11:29.343188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.343175 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f8c4903d-3833-40e5-88ee-d0369d2d4092" containerName="revoke-keys" Nov 01 23:11:29.343234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.343190 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c4903d-3833-40e5-88ee-d0369d2d4092" containerName="revoke-keys" Nov 01 23:11:29.343234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.343209 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1ecad335-0e3c-4911-bb34-3e8ab8d516fc" containerName="revoke-keys" Nov 01 23:11:29.343234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.343214 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecad335-0e3c-4911-bb34-3e8ab8d516fc" containerName="revoke-keys" Nov 01 23:11:29.343349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.343329 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c4903d-3833-40e5-88ee-d0369d2d4092" containerName="revoke-keys" Nov 01 23:11:29.343349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.343340 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecad335-0e3c-4911-bb34-3e8ab8d516fc" containerName="revoke-keys" Nov 01 23:11:29.347985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.347965 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.356504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.356473 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:11:29.356663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.356563 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:11:29.357717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.357704 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:11:29.357841 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.357815 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:11:29.371571 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.371546 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l"] Nov 01 23:11:29.454458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.454425 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.454642 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.454550 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.454697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.454686 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sldn\" (UniqueName: \"kubernetes.io/projected/36da46bf-f910-44ac-8dbf-f966a4a61798-kube-api-access-5sldn\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.555606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.555505 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.555760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.555615 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.555760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.555704 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sldn\" (UniqueName: \"kubernetes.io/projected/36da46bf-f910-44ac-8dbf-f966a4a61798-kube-api-access-5sldn\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.556328 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.556309 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.557892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.557875 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.567122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.567101 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sldn\" (UniqueName: \"kubernetes.io/projected/36da46bf-f910-44ac-8dbf-f966a4a61798-kube-api-access-5sldn\") pod \"maas-api-revoke-sub-75c6ea04-vgx8l\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.579790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.579766 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:29.661664 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.661635 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:29.694380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.694349 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:29.898222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.898136 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:29.940805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:29.940773 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l"] Nov 01 23:11:29.940927 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:29.940784 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36da46bf_f910_44ac_8dbf_f966a4a61798.slice/crio-9b6c140ca6e63577ec9ae14e2c37c279545123ca39d8f74ecaf673e9dece297a WatchSource:0}: Error finding container 9b6c140ca6e63577ec9ae14e2c37c279545123ca39d8f74ecaf673e9dece297a: Status 404 returned error can't find the container with id 9b6c140ca6e63577ec9ae14e2c37c279545123ca39d8f74ecaf673e9dece297a Nov 01 23:11:30.146043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.146011 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:30.295265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.295230 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:30.436295 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.436263 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc"] Nov 01 23:11:30.440618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.440592 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.449180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.449102 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc"] Nov 01 23:11:30.582224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.582187 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.582224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.582225 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.582481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.582249 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt2p4\" (UniqueName: \"kubernetes.io/projected/5affcc59-c738-44ad-a221-9b452f780e8b-kube-api-access-vt2p4\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.683636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.683594 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.683636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.683639 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.683978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.683665 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt2p4\" (UniqueName: \"kubernetes.io/projected/5affcc59-c738-44ad-a221-9b452f780e8b-kube-api-access-vt2p4\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.684441 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.684414 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.686424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.686395 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.692310 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.692279 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt2p4\" (UniqueName: \"kubernetes.io/projected/5affcc59-c738-44ad-a221-9b452f780e8b-kube-api-access-vt2p4\") pod \"maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.759349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.759291 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:30.891098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.891057 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" event={"ID":"36da46bf-f910-44ac-8dbf-f966a4a61798","Type":"ContainerStarted","Data":"45e8747e9a43077d9d8e826cfd27928bac47d128934dc60751a6bb50469bf34f"} Nov 01 23:11:30.891268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.891118 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" event={"ID":"36da46bf-f910-44ac-8dbf-f966a4a61798","Type":"ContainerStarted","Data":"9b6c140ca6e63577ec9ae14e2c37c279545123ca39d8f74ecaf673e9dece297a"} Nov 01 23:11:30.910465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:30.910417 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" podStartSLOduration=1.910404505 podStartE2EDuration="1.910404505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:30.908377826 +0000 UTC m=+2856.336255758" watchObservedRunningTime="2026-11-01 23:11:30.910404505 +0000 UTC m=+2856.338282456" Nov 01 23:11:31.040721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.040690 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc"] Nov 01 23:11:31.040866 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:31.040748 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5affcc59_c738_44ad_a221_9b452f780e8b.slice/crio-acb48f8a3abbe90d507876f1bb4ad6c92280145ab2b868dce3a5a94b46b68084 WatchSource:0}: Error finding container acb48f8a3abbe90d507876f1bb4ad6c92280145ab2b868dce3a5a94b46b68084: Status 404 returned error can't find the container with id acb48f8a3abbe90d507876f1bb4ad6c92280145ab2b868dce3a5a94b46b68084 Nov 01 23:11:31.055312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.055286 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:31.853139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.853101 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:31.917653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.916995 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" event={"ID":"5affcc59-c738-44ad-a221-9b452f780e8b","Type":"ContainerStarted","Data":"fb2879ca89ce60cabce5b317bfe64bbf9ef861a6e7252078d49bd058eb6cc4a1"} Nov 01 23:11:31.917653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.917064 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" event={"ID":"5affcc59-c738-44ad-a221-9b452f780e8b","Type":"ContainerStarted","Data":"acb48f8a3abbe90d507876f1bb4ad6c92280145ab2b868dce3a5a94b46b68084"} Nov 01 23:11:31.923459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.922202 4981 generic.go:356] "Generic (PLEG): container finished" podID="36da46bf-f910-44ac-8dbf-f966a4a61798" containerID="45e8747e9a43077d9d8e826cfd27928bac47d128934dc60751a6bb50469bf34f" exitCode=0 Nov 01 23:11:31.923459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.922418 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" event={"ID":"36da46bf-f910-44ac-8dbf-f966a4a61798","Type":"ContainerDied","Data":"45e8747e9a43077d9d8e826cfd27928bac47d128934dc60751a6bb50469bf34f"} Nov 01 23:11:31.940377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:31.940314 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" podStartSLOduration=1.9402991699999999 podStartE2EDuration="1.94029917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:31.935523658 +0000 UTC m=+2857.363401588" watchObservedRunningTime="2026-11-01 23:11:31.94029917 +0000 UTC m=+2857.368177103" Nov 01 23:11:32.257285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:32.257241 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" podUID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" containerName="payload-processing" containerID="cri-o://171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9" gracePeriod=25 Nov 01 23:11:32.271832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:32.271800 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" podUID="81ac8ace-f63d-4fb0-b642-5653bde50da7" containerName="payload-pre-processing" containerID="cri-o://f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8" gracePeriod=25 Nov 01 23:11:32.773709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:32.773676 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:32.937345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:32.937294 4981 generic.go:356] "Generic (PLEG): container finished" podID="5affcc59-c738-44ad-a221-9b452f780e8b" containerID="fb2879ca89ce60cabce5b317bfe64bbf9ef861a6e7252078d49bd058eb6cc4a1" exitCode=0 Nov 01 23:11:32.938315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:32.937350 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" event={"ID":"5affcc59-c738-44ad-a221-9b452f780e8b","Type":"ContainerDied","Data":"fb2879ca89ce60cabce5b317bfe64bbf9ef861a6e7252078d49bd058eb6cc4a1"} Nov 01 23:11:33.197901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.197878 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:33.339386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.339355 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-openshift-ca\") pod \"36da46bf-f910-44ac-8dbf-f966a4a61798\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " Nov 01 23:11:33.339601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.339403 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36da46bf-f910-44ac-8dbf-f966a4a61798-kube-api-access-5sldn\" (UniqueName: \"kubernetes.io/projected/36da46bf-f910-44ac-8dbf-f966a4a61798-kube-api-access-5sldn\") pod \"36da46bf-f910-44ac-8dbf-f966a4a61798\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " Nov 01 23:11:33.339601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.339545 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-xks-ca\") pod \"36da46bf-f910-44ac-8dbf-f966a4a61798\" (UID: \"36da46bf-f910-44ac-8dbf-f966a4a61798\") " Nov 01 23:11:33.339806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.339777 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-openshift-ca" pod "36da46bf-f910-44ac-8dbf-f966a4a61798" (UID: "36da46bf-f910-44ac-8dbf-f966a4a61798"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:33.340095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.340071 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:33.341598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.341527 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36da46bf-f910-44ac-8dbf-f966a4a61798-kube-api-access-5sldn" pod "36da46bf-f910-44ac-8dbf-f966a4a61798" (UID: "36da46bf-f910-44ac-8dbf-f966a4a61798"). InnerVolumeSpecName "kube-api-access-5sldn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:33.341598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.341556 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-xks-ca" pod "36da46bf-f910-44ac-8dbf-f966a4a61798" (UID: "36da46bf-f910-44ac-8dbf-f966a4a61798"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:33.441278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.441246 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36da46bf-f910-44ac-8dbf-f966a4a61798-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:33.441278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.441272 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sldn\" (UniqueName: \"kubernetes.io/projected/36da46bf-f910-44ac-8dbf-f966a4a61798-kube-api-access-5sldn\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:33.448883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.448851 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" podUID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:33.449021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.448879 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" podUID="81ac8ace-f63d-4fb0-b642-5653bde50da7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:33.713303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.713267 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:33.720448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.720414 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" podUID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:33.947503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.947476 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" Nov 01 23:11:33.948104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.947512 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l" event={"ID":"36da46bf-f910-44ac-8dbf-f966a4a61798","Type":"ContainerDied","Data":"9b6c140ca6e63577ec9ae14e2c37c279545123ca39d8f74ecaf673e9dece297a"} Nov 01 23:11:33.948104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:33.947559 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b6c140ca6e63577ec9ae14e2c37c279545123ca39d8f74ecaf673e9dece297a" Nov 01 23:11:34.093109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.093081 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:34.250793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.250753 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5affcc59-c738-44ad-a221-9b452f780e8b-kube-api-access-vt2p4\" (UniqueName: \"kubernetes.io/projected/5affcc59-c738-44ad-a221-9b452f780e8b-kube-api-access-vt2p4\") pod \"5affcc59-c738-44ad-a221-9b452f780e8b\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " Nov 01 23:11:34.250793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.250804 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-xks-ca\") pod \"5affcc59-c738-44ad-a221-9b452f780e8b\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " Nov 01 23:11:34.250793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.250841 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-openshift-ca\") pod \"5affcc59-c738-44ad-a221-9b452f780e8b\" (UID: \"5affcc59-c738-44ad-a221-9b452f780e8b\") " Nov 01 23:11:34.251267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.251243 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-openshift-ca" pod "5affcc59-c738-44ad-a221-9b452f780e8b" (UID: "5affcc59-c738-44ad-a221-9b452f780e8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:34.251482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.251468 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:34.253002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.252946 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5affcc59-c738-44ad-a221-9b452f780e8b-kube-api-access-vt2p4" pod "5affcc59-c738-44ad-a221-9b452f780e8b" (UID: "5affcc59-c738-44ad-a221-9b452f780e8b"). InnerVolumeSpecName "kube-api-access-vt2p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:34.253098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.253039 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-xks-ca" pod "5affcc59-c738-44ad-a221-9b452f780e8b" (UID: "5affcc59-c738-44ad-a221-9b452f780e8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:34.353220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.353189 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt2p4\" (UniqueName: \"kubernetes.io/projected/5affcc59-c738-44ad-a221-9b452f780e8b-kube-api-access-vt2p4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:34.353220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.353217 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5affcc59-c738-44ad-a221-9b452f780e8b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:34.453477 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.453448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:34.963879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.963816 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" event={"ID":"5affcc59-c738-44ad-a221-9b452f780e8b","Type":"ContainerDied","Data":"acb48f8a3abbe90d507876f1bb4ad6c92280145ab2b868dce3a5a94b46b68084"} Nov 01 23:11:34.964372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.963932 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acb48f8a3abbe90d507876f1bb4ad6c92280145ab2b868dce3a5a94b46b68084" Nov 01 23:11:34.964372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:34.964014 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc" Nov 01 23:11:35.263954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:35.263919 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:35.285864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:35.285833 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7"] Nov 01 23:11:35.355992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:35.355957 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:36.135129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.135096 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:36.425719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.425691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:36.526511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.526480 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:36.732220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.732135 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4"] Nov 01 23:11:36.732376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.732320 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" containerID="cri-o://46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6" gracePeriod=30 Nov 01 23:11:36.743765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.743737 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst"] Nov 01 23:11:36.743936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.743904 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" containerID="cri-o://7898651c56700954f3e68795eb8d8eee2693cab2b6c3a2c8528c089611e25d07" gracePeriod=30 Nov 01 23:11:36.806343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.806313 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:36.886017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:36.885988 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:37.131981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.131905 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l"] Nov 01 23:11:37.132125 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.132102 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" containerID="cri-o://a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e" gracePeriod=30 Nov 01 23:11:37.149158 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.149126 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:37.189850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.189815 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g"] Nov 01 23:11:37.190415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.190400 4981 cpu_manager.go:414] "containerMap: removing container" podUID="36da46bf-f910-44ac-8dbf-f966a4a61798" containerName="revoke-keys" Nov 01 23:11:37.190465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.190418 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36da46bf-f910-44ac-8dbf-f966a4a61798" containerName="revoke-keys" Nov 01 23:11:37.190503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.190474 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5affcc59-c738-44ad-a221-9b452f780e8b" containerName="revoke-keys" Nov 01 23:11:37.190503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.190480 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5affcc59-c738-44ad-a221-9b452f780e8b" containerName="revoke-keys" Nov 01 23:11:37.190612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.190566 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="36da46bf-f910-44ac-8dbf-f966a4a61798" containerName="revoke-keys" Nov 01 23:11:37.190612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.190607 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5affcc59-c738-44ad-a221-9b452f780e8b" containerName="revoke-keys" Nov 01 23:11:37.194430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.194414 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.199721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.199688 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:11:37.200186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.200157 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:11:37.207869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.205593 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g"] Nov 01 23:11:37.207869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.206041 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbr48\" (UniqueName: \"kubernetes.io/projected/8abfbdbc-0466-4b4e-9616-8eefca9f9884-kube-api-access-jbr48\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.207869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.206145 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.207869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.206309 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.292088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.292054 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:11:37.293264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.293191 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:11:37.307348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.307309 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.307552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.307500 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.309611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.307615 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbr48\" (UniqueName: \"kubernetes.io/projected/8abfbdbc-0466-4b4e-9616-8eefca9f9884-kube-api-access-jbr48\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.309611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.308937 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.312761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.312734 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.337683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.337659 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbr48\" (UniqueName: \"kubernetes.io/projected/8abfbdbc-0466-4b4e-9616-8eefca9f9884-kube-api-access-jbr48\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.589041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.589005 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:37.694512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.694473 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm"] Nov 01 23:11:37.701512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.701160 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.704869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.704828 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm"] Nov 01 23:11:37.712418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.712390 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.712646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.712624 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.712825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.712806 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2cw\" (UniqueName: \"kubernetes.io/projected/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-kube-api-access-8b2cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.814438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.814397 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.814665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.814483 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2cw\" (UniqueName: \"kubernetes.io/projected/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-kube-api-access-8b2cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.814878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.814842 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.815462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.815439 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.816984 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.816965 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.823442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.823419 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2cw\" (UniqueName: \"kubernetes.io/projected/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-kube-api-access-8b2cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:37.877531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.877504 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g"] Nov 01 23:11:37.877718 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:37.877687 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abfbdbc_0466_4b4e_9616_8eefca9f9884.slice/crio-72d56207f5e883c9c22818fec8b3e7a1b72e67664c7d638d030c4e111a307fd0 WatchSource:0}: Error finding container 72d56207f5e883c9c22818fec8b3e7a1b72e67664c7d638d030c4e111a307fd0: Status 404 returned error can't find the container with id 72d56207f5e883c9c22818fec8b3e7a1b72e67664c7d638d030c4e111a307fd0 Nov 01 23:11:37.899255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.899227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:37.992245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:37.992211 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" event={"ID":"8abfbdbc-0466-4b4e-9616-8eefca9f9884","Type":"ContainerStarted","Data":"72d56207f5e883c9c22818fec8b3e7a1b72e67664c7d638d030c4e111a307fd0"} Nov 01 23:11:38.023120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:38.023087 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:38.305833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:38.305799 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm"] Nov 01 23:11:38.682799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:38.682764 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:39.001947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:39.001913 4981 generic.go:356] "Generic (PLEG): container finished" podID="8abfbdbc-0466-4b4e-9616-8eefca9f9884" containerID="1975ca0aa780c22af9932c11c03d12d2dc9b6dd5a68dfc37e3ba8219f7ca2eb8" exitCode=0 Nov 01 23:11:39.002163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:39.001991 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" event={"ID":"8abfbdbc-0466-4b4e-9616-8eefca9f9884","Type":"ContainerDied","Data":"1975ca0aa780c22af9932c11c03d12d2dc9b6dd5a68dfc37e3ba8219f7ca2eb8"} Nov 01 23:11:39.003752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:39.003724 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" event={"ID":"18ffdcd5-d782-41d5-bbd7-92a219ff64e7","Type":"ContainerStarted","Data":"6fa44991b973164900d4a2dbe4dddba34466ad761e27975a51b5751a769c8e6a"} Nov 01 23:11:39.003873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:39.003771 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" event={"ID":"18ffdcd5-d782-41d5-bbd7-92a219ff64e7","Type":"ContainerStarted","Data":"641098e14104fb8d4d392c6c8adb0a6b6950dff22e93cead6268ca8bab927e4b"} Nov 01 23:11:39.040092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:39.040042 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" podStartSLOduration=2.040026706 podStartE2EDuration="2.040026706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:39.03925618 +0000 UTC m=+2864.467134113" watchObservedRunningTime="2026-11-01 23:11:39.040026706 +0000 UTC m=+2864.467904637" Nov 01 23:11:39.452384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:39.452308 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:39.971162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:39.971131 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:40.020132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.018456 4981 generic.go:356] "Generic (PLEG): container finished" podID="18ffdcd5-d782-41d5-bbd7-92a219ff64e7" containerID="6fa44991b973164900d4a2dbe4dddba34466ad761e27975a51b5751a769c8e6a" exitCode=0 Nov 01 23:11:40.020132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.019439 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" event={"ID":"18ffdcd5-d782-41d5-bbd7-92a219ff64e7","Type":"ContainerDied","Data":"6fa44991b973164900d4a2dbe4dddba34466ad761e27975a51b5751a769c8e6a"} Nov 01 23:11:40.210357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.210333 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:40.263677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.263515 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abfbdbc-0466-4b4e-9616-8eefca9f9884-kube-api-access-jbr48\" (UniqueName: \"kubernetes.io/projected/8abfbdbc-0466-4b4e-9616-8eefca9f9884-kube-api-access-jbr48\") pod \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " Nov 01 23:11:40.263902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.263697 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-openshift-ca\") pod \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " Nov 01 23:11:40.263902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.263776 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-xks-ca\") pod \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\" (UID: \"8abfbdbc-0466-4b4e-9616-8eefca9f9884\") " Nov 01 23:11:40.266029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.266000 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abfbdbc-0466-4b4e-9616-8eefca9f9884-kube-api-access-jbr48" pod "8abfbdbc-0466-4b4e-9616-8eefca9f9884" (UID: "8abfbdbc-0466-4b4e-9616-8eefca9f9884"). InnerVolumeSpecName "kube-api-access-jbr48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:40.266470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.266444 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-xks-ca" pod "8abfbdbc-0466-4b4e-9616-8eefca9f9884" (UID: "8abfbdbc-0466-4b4e-9616-8eefca9f9884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:40.266470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.266443 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-openshift-ca" pod "8abfbdbc-0466-4b4e-9616-8eefca9f9884" (UID: "8abfbdbc-0466-4b4e-9616-8eefca9f9884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:40.267383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.267363 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:40.365764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.365733 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbr48\" (UniqueName: \"kubernetes.io/projected/8abfbdbc-0466-4b4e-9616-8eefca9f9884-kube-api-access-jbr48\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:40.365764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.365759 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:40.365764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.365769 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abfbdbc-0466-4b4e-9616-8eefca9f9884-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:40.618450 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.618364 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn"] Nov 01 23:11:40.673240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.673200 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:40.771406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:40.771377 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:41.028646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.028613 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" Nov 01 23:11:41.028646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.028631 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g" event={"ID":"8abfbdbc-0466-4b4e-9616-8eefca9f9884","Type":"ContainerDied","Data":"72d56207f5e883c9c22818fec8b3e7a1b72e67664c7d638d030c4e111a307fd0"} Nov 01 23:11:41.028887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.028683 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72d56207f5e883c9c22818fec8b3e7a1b72e67664c7d638d030c4e111a307fd0" Nov 01 23:11:41.073955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.073885 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:41.198057 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.198033 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:41.277107 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.277075 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-openshift-ca\") pod \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " Nov 01 23:11:41.277309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.277172 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-xks-ca\") pod \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " Nov 01 23:11:41.277309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.277266 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-kube-api-access-8b2cw\" (UniqueName: \"kubernetes.io/projected/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-kube-api-access-8b2cw\") pod \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\" (UID: \"18ffdcd5-d782-41d5-bbd7-92a219ff64e7\") " Nov 01 23:11:41.277492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.277465 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-openshift-ca" pod "18ffdcd5-d782-41d5-bbd7-92a219ff64e7" (UID: "18ffdcd5-d782-41d5-bbd7-92a219ff64e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:41.277868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.277848 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:41.279410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.279352 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-kube-api-access-8b2cw" pod "18ffdcd5-d782-41d5-bbd7-92a219ff64e7" (UID: "18ffdcd5-d782-41d5-bbd7-92a219ff64e7"). InnerVolumeSpecName "kube-api-access-8b2cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:41.279410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.279353 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-xks-ca" pod "18ffdcd5-d782-41d5-bbd7-92a219ff64e7" (UID: "18ffdcd5-d782-41d5-bbd7-92a219ff64e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:41.379123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.379088 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:41.379123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.379122 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b2cw\" (UniqueName: \"kubernetes.io/projected/18ffdcd5-d782-41d5-bbd7-92a219ff64e7-kube-api-access-8b2cw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:41.876303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:41.876255 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:42.000850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.000827 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:11:42.043487 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.043444 4981 generic.go:356] "Generic (PLEG): container finished" podID="1563155b-37e1-4023-a721-545a505f7f6f" containerID="7898651c56700954f3e68795eb8d8eee2693cab2b6c3a2c8528c089611e25d07" exitCode=0 Nov 01 23:11:42.043730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.043661 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" event={"ID":"1563155b-37e1-4023-a721-545a505f7f6f","Type":"ContainerDied","Data":"7898651c56700954f3e68795eb8d8eee2693cab2b6c3a2c8528c089611e25d07"} Nov 01 23:11:42.045305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.045278 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" event={"ID":"18ffdcd5-d782-41d5-bbd7-92a219ff64e7","Type":"ContainerDied","Data":"641098e14104fb8d4d392c6c8adb0a6b6950dff22e93cead6268ca8bab927e4b"} Nov 01 23:11:42.045448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.045321 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641098e14104fb8d4d392c6c8adb0a6b6950dff22e93cead6268ca8bab927e4b" Nov 01 23:11:42.045448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.045367 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm" Nov 01 23:11:42.047653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.047626 4981 generic.go:356] "Generic (PLEG): container finished" podID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerID="46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6" exitCode=0 Nov 01 23:11:42.047774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.047681 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" Nov 01 23:11:42.047857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.047687 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" event={"ID":"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32","Type":"ContainerDied","Data":"46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6"} Nov 01 23:11:42.047857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.047818 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4" event={"ID":"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32","Type":"ContainerDied","Data":"c0d1c72f70e7d3109eebb73e72dcc131d170d15a19d4e6ebf67835c62a3b8a55"} Nov 01 23:11:42.047857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.047851 4981 scope.go:122] "RemoveContainer" containerID="46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6" Nov 01 23:11:42.072999 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.072971 4981 scope.go:122] "RemoveContainer" containerID="46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6" Nov 01 23:11:42.073461 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:42.073407 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6\": container with ID starting with 46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6 not found: ID does not exist" containerID="46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6" Nov 01 23:11:42.073461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.073446 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6"} err="failed to get container status \"46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6\": rpc error: code = NotFound desc = could not find container \"46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6\": container with ID starting with 46fc928bfeb9f56aa8d9e348560db3f472b5b6ff925911c883b257e9be03deb6 not found: ID does not exist" Nov 01 23:11:42.084447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.084426 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:11:42.089850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.089830 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istiod-ca-cert\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.089946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.089866 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-token\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-token\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.089946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.089905 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-certs\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.089946 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.089928 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-socket\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.090082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.089950 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-credential-socket\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.090129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090105 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-podinfo\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.090180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090160 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-envoy\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.090245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090184 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-certs" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.090245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090201 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-socket" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.090245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090191 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-kube-api-access-pxhb6\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-kube-api-access-pxhb6\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.090425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090276 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istiod-ca-cert" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.090425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090288 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-credential-socket" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.090425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090304 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-data\") pod \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\" (UID: \"1f614f4f-ff33-4e87-b4fd-5320b7b7ed32\") " Nov 01 23:11:42.090637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090600 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-data" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.090937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090916 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.091003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090942 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.091003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090956 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.091003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090970 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.091003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.090983 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.092569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.092543 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-token" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.092569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.092559 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-podinfo" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:42.092751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.092700 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-envoy" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.092751 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.092733 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-kube-api-access-pxhb6" pod "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" (UID: "1f614f4f-ff33-4e87-b4fd-5320b7b7ed32"). InnerVolumeSpecName "kube-api-access-pxhb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.192373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192340 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1563155b-37e1-4023-a721-545a505f7f6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563155b-37e1-4023-a721-545a505f7f6f-istio-podinfo\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192485 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-socket\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192549 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-credential-socket\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192601 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-istio-token\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-istio-token\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192660 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-data\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192691 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1563155b-37e1-4023-a721-545a505f7f6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563155b-37e1-4023-a721-545a505f7f6f-istiod-ca-cert\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192735 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-kube-api-access-cq2sx\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-kube-api-access-cq2sx\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192803 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-envoy\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.192919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192840 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-credential-socket" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.192919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192855 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-socket" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.192919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.192880 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-certs\") pod \"1563155b-37e1-4023-a721-545a505f7f6f\" (UID: \"1563155b-37e1-4023-a721-545a505f7f6f\") " Nov 01 23:11:42.193155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193035 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-data" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.193324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193262 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-certs" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193603 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1563155b-37e1-4023-a721-545a505f7f6f-istiod-ca-cert" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193788 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193813 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193835 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193849 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1563155b-37e1-4023-a721-545a505f7f6f-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193863 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193877 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193890 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193905 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxhb6\" (UniqueName: \"kubernetes.io/projected/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32-kube-api-access-pxhb6\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.194162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.193920 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.195017 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.194981 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1563155b-37e1-4023-a721-545a505f7f6f-istio-podinfo" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:42.195331 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.195287 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-kube-api-access-cq2sx" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "kube-api-access-cq2sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.195612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.195588 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-envoy" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.195837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.195820 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-istio-token" pod "1563155b-37e1-4023-a721-545a505f7f6f" (UID: "1563155b-37e1-4023-a721-545a505f7f6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.295109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.295072 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1563155b-37e1-4023-a721-545a505f7f6f-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.295277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.295115 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.295277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.295131 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq2sx\" (UniqueName: \"kubernetes.io/projected/1563155b-37e1-4023-a721-545a505f7f6f-kube-api-access-cq2sx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.295277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.295142 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1563155b-37e1-4023-a721-545a505f7f6f-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.379015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.378984 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4"] Nov 01 23:11:42.382791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.382725 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-tk5s4"] Nov 01 23:11:42.409769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.409740 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:11:42.497181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497149 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-socket\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497222 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-certs\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497243 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-credential-socket\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497275 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-kube-api-access-b6wdw\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-kube-api-access-b6wdw\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497341 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istiod-ca-cert\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497365 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-envoy\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497413 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-data\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497508 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-socket" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.497731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497552 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-podinfo\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.497731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497566 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-certs" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.497731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497627 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-credential-socket" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.497731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497637 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-token\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-token\") pod \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\" (UID: \"7c914dd3-3527-40a0-8dfc-a2a884e0f49b\") " Nov 01 23:11:42.498030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.497921 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istiod-ca-cert" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:42.498347 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.498299 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.498492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.498351 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.498492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.498367 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.498656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.498491 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.498656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.498494 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-data" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.500672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.500645 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-envoy" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:42.500828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.500727 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-token" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.500901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.500834 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-podinfo" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:11:42.505183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.505126 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-kube-api-access-b6wdw" pod "7c914dd3-3527-40a0-8dfc-a2a884e0f49b" (UID: "7c914dd3-3527-40a0-8dfc-a2a884e0f49b"). InnerVolumeSpecName "kube-api-access-b6wdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:42.599645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.599614 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.599645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.599642 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.599859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.599653 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.599859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.599664 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.599859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.599675 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6wdw\" (UniqueName: \"kubernetes.io/projected/7c914dd3-3527-40a0-8dfc-a2a884e0f49b-kube-api-access-b6wdw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:42.737412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.737380 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" path="/var/lib/kubelet/pods/1f614f4f-ff33-4e87-b4fd-5320b7b7ed32/volumes" Nov 01 23:11:42.858368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:42.858291 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:43.057053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.057020 4981 generic.go:356] "Generic (PLEG): container finished" podID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerID="a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e" exitCode=0 Nov 01 23:11:43.057551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.057068 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" Nov 01 23:11:43.057551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.057112 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" event={"ID":"7c914dd3-3527-40a0-8dfc-a2a884e0f49b","Type":"ContainerDied","Data":"a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e"} Nov 01 23:11:43.057551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.057183 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l" event={"ID":"7c914dd3-3527-40a0-8dfc-a2a884e0f49b","Type":"ContainerDied","Data":"dd1e5c228188e309857eb9302a1ad4eb2e28a9a101c7b879b978a1a09257c1fd"} Nov 01 23:11:43.057551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.057218 4981 scope.go:122] "RemoveContainer" containerID="a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e" Nov 01 23:11:43.058789 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.058768 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" Nov 01 23:11:43.058941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.058805 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst" event={"ID":"1563155b-37e1-4023-a721-545a505f7f6f","Type":"ContainerDied","Data":"60e1ab35b49b69891036be4ff4ec2fa4f79014b4eed8d74b59086778b8bbccc0"} Nov 01 23:11:43.072103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.072086 4981 scope.go:122] "RemoveContainer" containerID="a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e" Nov 01 23:11:43.072371 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:43.072350 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e\": container with ID starting with a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e not found: ID does not exist" containerID="a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e" Nov 01 23:11:43.072466 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.072379 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e"} err="failed to get container status \"a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e\": rpc error: code = NotFound desc = could not find container \"a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e\": container with ID starting with a0ca2d51d90d70330705e8a16994be2249521a13254317ec2c4f22081757918e not found: ID does not exist" Nov 01 23:11:43.072466 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.072395 4981 scope.go:122] "RemoveContainer" containerID="7898651c56700954f3e68795eb8d8eee2693cab2b6c3a2c8528c089611e25d07" Nov 01 23:11:43.081927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.081903 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l"] Nov 01 23:11:43.085691 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.085672 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a9f94e-openshift-default-7f7f7978f-hnf8l"] Nov 01 23:11:43.094942 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.094920 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst"] Nov 01 23:11:43.099745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.099722 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-7366bf-openshift-default-84c857d6b5-8crst"] Nov 01 23:11:43.448814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.448774 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" podUID="81ac8ace-f63d-4fb0-b642-5653bde50da7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:43.448814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.448793 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" podUID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:43.720322 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.720236 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" podUID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:11:43.846402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:43.846371 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:44.479491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.479456 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:44.525830 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:44.525767 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:11:44.578176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.578145 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:44.670735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.670695 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk"] Nov 01 23:11:44.671373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671351 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671376 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671406 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671414 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671429 4981 cpu_manager.go:414] "containerMap: removing container" podUID="18ffdcd5-d782-41d5-bbd7-92a219ff64e7" containerName="revoke-keys" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671437 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ffdcd5-d782-41d5-bbd7-92a219ff64e7" containerName="revoke-keys" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671465 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671472 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" Nov 01 23:11:44.671504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671503 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8abfbdbc-0466-4b4e-9616-8eefca9f9884" containerName="revoke-keys" Nov 01 23:11:44.671981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671510 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abfbdbc-0466-4b4e-9616-8eefca9f9884" containerName="revoke-keys" Nov 01 23:11:44.671981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671685 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1563155b-37e1-4023-a721-545a505f7f6f" containerName="istio-proxy" Nov 01 23:11:44.671981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671704 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abfbdbc-0466-4b4e-9616-8eefca9f9884" containerName="revoke-keys" Nov 01 23:11:44.671981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671716 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" containerName="istio-proxy" Nov 01 23:11:44.671981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671730 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f614f4f-ff33-4e87-b4fd-5320b7b7ed32" containerName="istio-proxy" Nov 01 23:11:44.671981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.671739 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ffdcd5-d782-41d5-bbd7-92a219ff64e7" containerName="revoke-keys" Nov 01 23:11:44.677478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.677447 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.681147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.681124 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:11:44.682248 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.682227 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:11:44.686506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.686481 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk"] Nov 01 23:11:44.735128 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.735039 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.735294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.735150 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.735294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.735257 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82h8p\" (UniqueName: \"kubernetes.io/projected/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-kube-api-access-82h8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.736586 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.736551 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1563155b-37e1-4023-a721-545a505f7f6f" path="/var/lib/kubelet/pods/1563155b-37e1-4023-a721-545a505f7f6f/volumes" Nov 01 23:11:44.737076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.737060 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c914dd3-3527-40a0-8dfc-a2a884e0f49b" path="/var/lib/kubelet/pods/7c914dd3-3527-40a0-8dfc-a2a884e0f49b/volumes" Nov 01 23:11:44.753096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.753071 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:11:44.753227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.753157 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:11:44.837001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.836966 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.837185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.837033 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.837185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.837090 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82h8p\" (UniqueName: \"kubernetes.io/projected/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-kube-api-access-82h8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.837678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.837655 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.839538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.839517 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.845354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.845324 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82h8p\" (UniqueName: \"kubernetes.io/projected/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-kube-api-access-82h8p\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:44.998041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:44.997956 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:45.284763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.284731 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk"] Nov 01 23:11:45.284897 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:45.284811 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea539f1_ccaf_4be4_94ce_25bf78db0a2a.slice/crio-24a2f10d9eac59b87132bb10fba29707cf61cce5e4b1c46d9fd61de98380f5d0 WatchSource:0}: Error finding container 24a2f10d9eac59b87132bb10fba29707cf61cce5e4b1c46d9fd61de98380f5d0: Status 404 returned error can't find the container with id 24a2f10d9eac59b87132bb10fba29707cf61cce5e4b1c46d9fd61de98380f5d0 Nov 01 23:11:45.466364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.466324 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:45.618802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.618731 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" podUID="42a1952a-0a95-40f4-8a94-d7580b97f74c" containerName="payload-processing" containerID="cri-o://505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7" gracePeriod=25 Nov 01 23:11:45.885551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.885524 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 23:11:45.959431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.959396 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-kube-api-access-zjdnr\" (UniqueName: \"kubernetes.io/projected/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-kube-api-access-zjdnr\") pod \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " Nov 01 23:11:45.959656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.959609 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-plugins-config-volume\") pod \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\" (UID: \"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438\") " Nov 01 23:11:45.960758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.960729 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-plugins-config-volume" pod "8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" (UID: "8f6ddf6f-99e9-4569-ba8a-f79b1dfef438"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:45.963771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.962868 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-kube-api-access-zjdnr" pod "8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" (UID: "8f6ddf6f-99e9-4569-ba8a-f79b1dfef438"). InnerVolumeSpecName "kube-api-access-zjdnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:45.969311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:45.969289 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 23:11:46.060542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.060509 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a1952a-0a95-40f4-8a94-d7580b97f74c-kube-api-access-dtf8q\" (UniqueName: \"kubernetes.io/projected/42a1952a-0a95-40f4-8a94-d7580b97f74c-kube-api-access-dtf8q\") pod \"42a1952a-0a95-40f4-8a94-d7580b97f74c\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " Nov 01 23:11:46.060542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.060548 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42a1952a-0a95-40f4-8a94-d7580b97f74c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a1952a-0a95-40f4-8a94-d7580b97f74c-plugins-config-volume\") pod \"42a1952a-0a95-40f4-8a94-d7580b97f74c\" (UID: \"42a1952a-0a95-40f4-8a94-d7580b97f74c\") " Nov 01 23:11:46.060954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.060929 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a1952a-0a95-40f4-8a94-d7580b97f74c-plugins-config-volume" pod "42a1952a-0a95-40f4-8a94-d7580b97f74c" (UID: "42a1952a-0a95-40f4-8a94-d7580b97f74c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:46.061139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.061118 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42a1952a-0a95-40f4-8a94-d7580b97f74c-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:46.061228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.061142 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:46.061228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.061151 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjdnr\" (UniqueName: \"kubernetes.io/projected/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438-kube-api-access-zjdnr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:46.062747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.062727 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a1952a-0a95-40f4-8a94-d7580b97f74c-kube-api-access-dtf8q" pod "42a1952a-0a95-40f4-8a94-d7580b97f74c" (UID: "42a1952a-0a95-40f4-8a94-d7580b97f74c"). InnerVolumeSpecName "kube-api-access-dtf8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:46.090087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.090056 4981 generic.go:356] "Generic (PLEG): container finished" podID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" containerID="f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac" exitCode=0 Nov 01 23:11:46.090300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.090110 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" Nov 01 23:11:46.090300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.090142 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" event={"ID":"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438","Type":"ContainerDied","Data":"f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac"} Nov 01 23:11:46.090300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.090203 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn" event={"ID":"8f6ddf6f-99e9-4569-ba8a-f79b1dfef438","Type":"ContainerDied","Data":"fba2d6c4496ce7f4b08bd8693ad9df66c5e25ab7f84ce12fbdb2673bad6c2b3f"} Nov 01 23:11:46.090300 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.090234 4981 scope.go:122] "RemoveContainer" containerID="f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac" Nov 01 23:11:46.091777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.091755 4981 generic.go:356] "Generic (PLEG): container finished" podID="42a1952a-0a95-40f4-8a94-d7580b97f74c" containerID="505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7" exitCode=0 Nov 01 23:11:46.091873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.091795 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" Nov 01 23:11:46.091873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.091832 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" event={"ID":"42a1952a-0a95-40f4-8a94-d7580b97f74c","Type":"ContainerDied","Data":"505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7"} Nov 01 23:11:46.091969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.091889 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn" event={"ID":"42a1952a-0a95-40f4-8a94-d7580b97f74c","Type":"ContainerDied","Data":"484287c36267148270e7b55a31c0da1634c82e1acf19d76eeb8392d1149e0250"} Nov 01 23:11:46.094009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.093983 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" event={"ID":"cea539f1-ccaf-4be4-94ce-25bf78db0a2a","Type":"ContainerStarted","Data":"4c31314af1692dc7f23ba44f8731e237e3502089c5d1ec2a5c112037af86b4dc"} Nov 01 23:11:46.094127 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.094031 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" event={"ID":"cea539f1-ccaf-4be4-94ce-25bf78db0a2a","Type":"ContainerStarted","Data":"24a2f10d9eac59b87132bb10fba29707cf61cce5e4b1c46d9fd61de98380f5d0"} Nov 01 23:11:46.102806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.102759 4981 scope.go:122] "RemoveContainer" containerID="f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac" Nov 01 23:11:46.103108 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:46.103084 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac\": container with ID starting with f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac not found: ID does not exist" containerID="f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac" Nov 01 23:11:46.103195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.103115 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac"} err="failed to get container status \"f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac\": rpc error: code = NotFound desc = could not find container \"f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac\": container with ID starting with f2960ecd65e08e15d7a330f28d826ac3de8e3b9ff8a7cc8a3476c47c070e6eac not found: ID does not exist" Nov 01 23:11:46.103195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.103131 4981 scope.go:122] "RemoveContainer" containerID="505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7" Nov 01 23:11:46.111497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.111475 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5"] Nov 01 23:11:46.112103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.112085 4981 cpu_manager.go:414] "containerMap: removing container" podUID="42a1952a-0a95-40f4-8a94-d7580b97f74c" containerName="payload-processing" Nov 01 23:11:46.112198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.112104 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a1952a-0a95-40f4-8a94-d7580b97f74c" containerName="payload-processing" Nov 01 23:11:46.112198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.112188 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" containerName="payload-pre-processing" Nov 01 23:11:46.112313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.112197 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" containerName="payload-pre-processing" Nov 01 23:11:46.112368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.112336 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" containerName="payload-pre-processing" Nov 01 23:11:46.112368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.112353 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a1952a-0a95-40f4-8a94-d7580b97f74c" containerName="payload-processing" Nov 01 23:11:46.116936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.116917 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.120924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.120903 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b"] Nov 01 23:11:46.125723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.125701 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.125852 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.125837 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5"] Nov 01 23:11:46.126782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.126748 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" podStartSLOduration=2.126737897 podStartE2EDuration="2.126737897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:46.119467001 +0000 UTC m=+2871.547344933" watchObservedRunningTime="2026-11-01 23:11:46.126737897 +0000 UTC m=+2871.554615828" Nov 01 23:11:46.140086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.140051 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b"] Nov 01 23:11:46.146972 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.146951 4981 scope.go:122] "RemoveContainer" containerID="505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7" Nov 01 23:11:46.149403 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:46.149372 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7\": container with ID starting with 505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7 not found: ID does not exist" containerID="505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7" Nov 01 23:11:46.149514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.149412 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7"} err="failed to get container status \"505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7\": rpc error: code = NotFound desc = could not find container \"505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7\": container with ID starting with 505582acfb3bd2177265a4dea902257bec6fbb6cd9890841ba41263ca8b152c7 not found: ID does not exist" Nov 01 23:11:46.150904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.150884 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn"] Nov 01 23:11:46.156880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.156856 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a9f94e-bdc6477dc-n9vjn"] Nov 01 23:11:46.167886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.167853 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtf8q\" (UniqueName: \"kubernetes.io/projected/42a1952a-0a95-40f4-8a94-d7580b97f74c-kube-api-access-dtf8q\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:46.181839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.181809 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn"] Nov 01 23:11:46.186755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.186731 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-f338e4-7977d7f87f-bvxvn"] Nov 01 23:11:46.269001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.268962 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.269182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.269034 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tcjg\" (UniqueName: \"kubernetes.io/projected/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-kube-api-access-9tcjg\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.269182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.269144 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a487583-0870-4da6-b967-5ec01928d764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.269328 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.269228 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a487583-0870-4da6-b967-5ec01928d764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.269328 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.269293 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hzn\" (UniqueName: \"kubernetes.io/projected/9a487583-0870-4da6-b967-5ec01928d764-kube-api-access-r5hzn\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.269458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.269395 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.371154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.371071 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a487583-0870-4da6-b967-5ec01928d764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.371154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.371144 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a487583-0870-4da6-b967-5ec01928d764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.371394 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.371195 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hzn\" (UniqueName: \"kubernetes.io/projected/9a487583-0870-4da6-b967-5ec01928d764-kube-api-access-r5hzn\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.371394 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.371224 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.371394 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.371371 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.371614 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.371427 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tcjg\" (UniqueName: \"kubernetes.io/projected/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-kube-api-access-9tcjg\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.371958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.371923 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a487583-0870-4da6-b967-5ec01928d764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.372146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.372114 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.373683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.373662 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a487583-0870-4da6-b967-5ec01928d764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.373772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.373686 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.384557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.384530 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:46.384950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.384929 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hzn\" (UniqueName: \"kubernetes.io/projected/9a487583-0870-4da6-b967-5ec01928d764-kube-api-access-r5hzn\") pod \"maas-api-revoke-sub-149987a3-hkph5\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.385522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.385504 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tcjg\" (UniqueName: \"kubernetes.io/projected/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-kube-api-access-9tcjg\") pod \"maas-api-revoke-sub-fe424e47-qg46b\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.464957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.464927 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:46.474217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.474193 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:46.735133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.735103 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a1952a-0a95-40f4-8a94-d7580b97f74c" path="/var/lib/kubelet/pods/42a1952a-0a95-40f4-8a94-d7580b97f74c/volumes" Nov 01 23:11:46.735560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.735545 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6ddf6f-99e9-4569-ba8a-f79b1dfef438" path="/var/lib/kubelet/pods/8f6ddf6f-99e9-4569-ba8a-f79b1dfef438/volumes" Nov 01 23:11:46.750140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.750112 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5"] Nov 01 23:11:46.750467 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:46.750442 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a487583_0870_4da6_b967_5ec01928d764.slice/crio-78d5e6000a39cc6a317fee650d3efffeeaf1b57c7e08bd1062e6a9a558513695 WatchSource:0}: Error finding container 78d5e6000a39cc6a317fee650d3efffeeaf1b57c7e08bd1062e6a9a558513695: Status 404 returned error can't find the container with id 78d5e6000a39cc6a317fee650d3efffeeaf1b57c7e08bd1062e6a9a558513695 Nov 01 23:11:46.772973 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:46.772943 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d3dcc1_b6ee_4998_9aba_1d76febe1de5.slice/crio-401c180d4a628f31afc8421cb82cf7c3341ef4a80577b5df12461d09392192f5 WatchSource:0}: Error finding container 401c180d4a628f31afc8421cb82cf7c3341ef4a80577b5df12461d09392192f5: Status 404 returned error can't find the container with id 401c180d4a628f31afc8421cb82cf7c3341ef4a80577b5df12461d09392192f5 Nov 01 23:11:46.774760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:46.774737 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b"] Nov 01 23:11:47.022743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.022630 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5"] Nov 01 23:11:47.026305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.026273 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb1fc58-9dcl5"] Nov 01 23:11:47.105610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.105543 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" event={"ID":"9a487583-0870-4da6-b967-5ec01928d764","Type":"ContainerStarted","Data":"78d5e6000a39cc6a317fee650d3efffeeaf1b57c7e08bd1062e6a9a558513695"} Nov 01 23:11:47.107152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.107127 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" event={"ID":"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5","Type":"ContainerStarted","Data":"401c180d4a628f31afc8421cb82cf7c3341ef4a80577b5df12461d09392192f5"} Nov 01 23:11:47.108644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.108621 4981 generic.go:356] "Generic (PLEG): container finished" podID="cea539f1-ccaf-4be4-94ce-25bf78db0a2a" containerID="4c31314af1692dc7f23ba44f8731e237e3502089c5d1ec2a5c112037af86b4dc" exitCode=0 Nov 01 23:11:47.108736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.108692 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" event={"ID":"cea539f1-ccaf-4be4-94ce-25bf78db0a2a","Type":"ContainerDied","Data":"4c31314af1692dc7f23ba44f8731e237e3502089c5d1ec2a5c112037af86b4dc"} Nov 01 23:11:47.187662 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.187632 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:47.417700 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.417664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:47.521399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.521362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:47.875908 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.875878 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:47.972418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:47.972385 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:48.071250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.071165 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:48.118201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.118159 4981 generic.go:356] "Generic (PLEG): container finished" podID="9a487583-0870-4da6-b967-5ec01928d764" containerID="d761416ba996277ba782eaf0d3ec6605001d0c7f8f8b294cd2d68602c88f0312" exitCode=0 Nov 01 23:11:48.118387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.118243 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" event={"ID":"9a487583-0870-4da6-b967-5ec01928d764","Type":"ContainerDied","Data":"d761416ba996277ba782eaf0d3ec6605001d0c7f8f8b294cd2d68602c88f0312"} Nov 01 23:11:48.119941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.119917 4981 generic.go:356] "Generic (PLEG): container finished" podID="a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" containerID="6dc5ebefbd68e08c0e089afd8632cad888423cda63f5746152fa14a567aeaf53" exitCode=0 Nov 01 23:11:48.120045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.119983 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" event={"ID":"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5","Type":"ContainerDied","Data":"6dc5ebefbd68e08c0e089afd8632cad888423cda63f5746152fa14a567aeaf53"} Nov 01 23:11:48.186606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.184119 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:48.315351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.315324 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:48.319665 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.319621 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g"] Nov 01 23:11:48.324895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.324829 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-075771-dlg4g"] Nov 01 23:11:48.331337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.331314 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:48.418796 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.418759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:48.519953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.519913 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-xks-ca\") pod \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " Nov 01 23:11:48.520140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.520070 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-openshift-ca\") pod \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " Nov 01 23:11:48.520140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.520135 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-kube-api-access-82h8p\" (UniqueName: \"kubernetes.io/projected/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-kube-api-access-82h8p\") pod \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\" (UID: \"cea539f1-ccaf-4be4-94ce-25bf78db0a2a\") " Nov 01 23:11:48.520413 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.520388 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-openshift-ca" pod "cea539f1-ccaf-4be4-94ce-25bf78db0a2a" (UID: "cea539f1-ccaf-4be4-94ce-25bf78db0a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:48.520555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.520538 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:48.522089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.522055 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-xks-ca" pod "cea539f1-ccaf-4be4-94ce-25bf78db0a2a" (UID: "cea539f1-ccaf-4be4-94ce-25bf78db0a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:48.522303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.522286 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-kube-api-access-82h8p" pod "cea539f1-ccaf-4be4-94ce-25bf78db0a2a" (UID: "cea539f1-ccaf-4be4-94ce-25bf78db0a2a"). InnerVolumeSpecName "kube-api-access-82h8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:48.622261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.622175 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:48.622261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.622212 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82h8p\" (UniqueName: \"kubernetes.io/projected/cea539f1-ccaf-4be4-94ce-25bf78db0a2a-kube-api-access-82h8p\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:48.735096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.735061 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a8d29e3-8c54-408d-9371-b0f2267b3908" path="/var/lib/kubelet/pods/5a8d29e3-8c54-408d-9371-b0f2267b3908/volumes" Nov 01 23:11:48.735522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.735507 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abfbdbc-0466-4b4e-9616-8eefca9f9884" path="/var/lib/kubelet/pods/8abfbdbc-0466-4b4e-9616-8eefca9f9884/volumes" Nov 01 23:11:48.847423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:48.847390 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:49.129571 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.129543 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" Nov 01 23:11:49.129571 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.129586 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk" event={"ID":"cea539f1-ccaf-4be4-94ce-25bf78db0a2a","Type":"ContainerDied","Data":"24a2f10d9eac59b87132bb10fba29707cf61cce5e4b1c46d9fd61de98380f5d0"} Nov 01 23:11:49.130074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.129630 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a2f10d9eac59b87132bb10fba29707cf61cce5e4b1c46d9fd61de98380f5d0" Nov 01 23:11:49.286339 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.286304 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:49.329214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.329189 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:49.431879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.431848 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-openshift-ca\") pod \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " Nov 01 23:11:49.432061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.431892 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a487583-0870-4da6-b967-5ec01928d764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a487583-0870-4da6-b967-5ec01928d764-maas-api-xks-ca\") pod \"9a487583-0870-4da6-b967-5ec01928d764\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " Nov 01 23:11:49.432122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.432106 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-xks-ca\") pod \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " Nov 01 23:11:49.432163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.432129 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-kube-api-access-9tcjg\" (UniqueName: \"kubernetes.io/projected/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-kube-api-access-9tcjg\") pod \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\" (UID: \"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5\") " Nov 01 23:11:49.432163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.432152 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a487583-0870-4da6-b967-5ec01928d764-kube-api-access-r5hzn\" (UniqueName: \"kubernetes.io/projected/9a487583-0870-4da6-b967-5ec01928d764-kube-api-access-r5hzn\") pod \"9a487583-0870-4da6-b967-5ec01928d764\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " Nov 01 23:11:49.432281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.432192 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a487583-0870-4da6-b967-5ec01928d764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a487583-0870-4da6-b967-5ec01928d764-maas-api-openshift-ca\") pod \"9a487583-0870-4da6-b967-5ec01928d764\" (UID: \"9a487583-0870-4da6-b967-5ec01928d764\") " Nov 01 23:11:49.432281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.432260 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-openshift-ca" pod "a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" (UID: "a0d3dcc1-b6ee-4998-9aba-1d76febe1de5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:49.432657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.432635 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:49.432657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.432641 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a487583-0870-4da6-b967-5ec01928d764-maas-api-openshift-ca" pod "9a487583-0870-4da6-b967-5ec01928d764" (UID: "9a487583-0870-4da6-b967-5ec01928d764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:49.434469 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.434443 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a487583-0870-4da6-b967-5ec01928d764-maas-api-xks-ca" pod "9a487583-0870-4da6-b967-5ec01928d764" (UID: "9a487583-0870-4da6-b967-5ec01928d764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:49.434752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.434720 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a487583-0870-4da6-b967-5ec01928d764-kube-api-access-r5hzn" pod "9a487583-0870-4da6-b967-5ec01928d764" (UID: "9a487583-0870-4da6-b967-5ec01928d764"). InnerVolumeSpecName "kube-api-access-r5hzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:49.434861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.434786 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-xks-ca" pod "a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" (UID: "a0d3dcc1-b6ee-4998-9aba-1d76febe1de5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:49.434861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.434801 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-kube-api-access-9tcjg" pod "a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" (UID: "a0d3dcc1-b6ee-4998-9aba-1d76febe1de5"). InnerVolumeSpecName "kube-api-access-9tcjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:49.533504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.533470 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:49.533504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.533501 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tcjg\" (UniqueName: \"kubernetes.io/projected/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5-kube-api-access-9tcjg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:49.533504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.533512 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hzn\" (UniqueName: \"kubernetes.io/projected/9a487583-0870-4da6-b967-5ec01928d764-kube-api-access-r5hzn\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:49.533770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.533526 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a487583-0870-4da6-b967-5ec01928d764-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:49.533770 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.533536 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a487583-0870-4da6-b967-5ec01928d764-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:49.541061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:49.541020 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:50.139175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.139146 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" Nov 01 23:11:50.139603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.139186 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b" event={"ID":"a0d3dcc1-b6ee-4998-9aba-1d76febe1de5","Type":"ContainerDied","Data":"401c180d4a628f31afc8421cb82cf7c3341ef4a80577b5df12461d09392192f5"} Nov 01 23:11:50.139603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.139233 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401c180d4a628f31afc8421cb82cf7c3341ef4a80577b5df12461d09392192f5" Nov 01 23:11:50.140892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.140868 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" event={"ID":"9a487583-0870-4da6-b967-5ec01928d764","Type":"ContainerDied","Data":"78d5e6000a39cc6a317fee650d3efffeeaf1b57c7e08bd1062e6a9a558513695"} Nov 01 23:11:50.141005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.140896 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5" Nov 01 23:11:50.141005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.140906 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d5e6000a39cc6a317fee650d3efffeeaf1b57c7e08bd1062e6a9a558513695" Nov 01 23:11:50.267145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.267116 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:50.308188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.308140 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" containerID="cri-o://836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b" gracePeriod=45 Nov 01 23:11:50.584454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.584430 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 23:11:50.747452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747364 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e8cc95-9482-47a6-b8c2-2507116356d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e8cc95-9482-47a6-b8c2-2507116356d4-tls-certs\") pod \"a2e8cc95-9482-47a6-b8c2-2507116356d4\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " Nov 01 23:11:50.747452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747442 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-home\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-home\") pod \"a2e8cc95-9482-47a6-b8c2-2507116356d4\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " Nov 01 23:11:50.747802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747596 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-tmp-dir\") pod \"a2e8cc95-9482-47a6-b8c2-2507116356d4\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " Nov 01 23:11:50.747802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747689 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-model-cache\") pod \"a2e8cc95-9482-47a6-b8c2-2507116356d4\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " Nov 01 23:11:50.747802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747739 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e8cc95-9482-47a6-b8c2-2507116356d4-kube-api-access-z5vqk\" (UniqueName: \"kubernetes.io/projected/a2e8cc95-9482-47a6-b8c2-2507116356d4-kube-api-access-z5vqk\") pod \"a2e8cc95-9482-47a6-b8c2-2507116356d4\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " Nov 01 23:11:50.747802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747759 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-dshm\") pod \"a2e8cc95-9482-47a6-b8c2-2507116356d4\" (UID: \"a2e8cc95-9482-47a6-b8c2-2507116356d4\") " Nov 01 23:11:50.747802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747762 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-home" pod "a2e8cc95-9482-47a6-b8c2-2507116356d4" (UID: "a2e8cc95-9482-47a6-b8c2-2507116356d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.748105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747939 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-model-cache" pod "a2e8cc95-9482-47a6-b8c2-2507116356d4" (UID: "a2e8cc95-9482-47a6-b8c2-2507116356d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.748105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.747959 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-tmp-dir" pod "a2e8cc95-9482-47a6-b8c2-2507116356d4" (UID: "a2e8cc95-9482-47a6-b8c2-2507116356d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.748234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.748215 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.748296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.748261 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.748296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.748277 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.750018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.749991 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-dshm" pod "a2e8cc95-9482-47a6-b8c2-2507116356d4" (UID: "a2e8cc95-9482-47a6-b8c2-2507116356d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:11:50.750137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.750103 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e8cc95-9482-47a6-b8c2-2507116356d4-tls-certs" pod "a2e8cc95-9482-47a6-b8c2-2507116356d4" (UID: "a2e8cc95-9482-47a6-b8c2-2507116356d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:11:50.750137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.750111 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e8cc95-9482-47a6-b8c2-2507116356d4-kube-api-access-z5vqk" pod "a2e8cc95-9482-47a6-b8c2-2507116356d4" (UID: "a2e8cc95-9482-47a6-b8c2-2507116356d4"). InnerVolumeSpecName "kube-api-access-z5vqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:50.849061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.849023 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5vqk\" (UniqueName: \"kubernetes.io/projected/a2e8cc95-9482-47a6-b8c2-2507116356d4-kube-api-access-z5vqk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.849061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.849054 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2e8cc95-9482-47a6-b8c2-2507116356d4-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.849061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.849063 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2e8cc95-9482-47a6-b8c2-2507116356d4-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:50.979560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:50.979520 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:51.150622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.150526 4981 generic.go:356] "Generic (PLEG): container finished" podID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerID="836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b" exitCode=0 Nov 01 23:11:51.150622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.150598 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" Nov 01 23:11:51.150622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.150606 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" event={"ID":"a2e8cc95-9482-47a6-b8c2-2507116356d4","Type":"ContainerDied","Data":"836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b"} Nov 01 23:11:51.151147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.150660 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7" event={"ID":"a2e8cc95-9482-47a6-b8c2-2507116356d4","Type":"ContainerDied","Data":"e8750972590f58fd72d4b24d9a698283b369da29af7352211734e45a51b5f65b"} Nov 01 23:11:51.151147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.150683 4981 scope.go:122] "RemoveContainer" containerID="836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b" Nov 01 23:11:51.161679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.161662 4981 scope.go:122] "RemoveContainer" containerID="836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b" Nov 01 23:11:51.161961 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:51.161940 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b\": container with ID starting with 836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b not found: ID does not exist" containerID="836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b" Nov 01 23:11:51.162042 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.161973 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b"} err="failed to get container status \"836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b\": rpc error: code = NotFound desc = could not find container \"836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b\": container with ID starting with 836932b6d5ab65f8c429ee80f1f5ba5a2c3fe6b1877cd65567eeaf17bc338b3b not found: ID does not exist" Nov 01 23:11:51.177262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.177234 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7"] Nov 01 23:11:51.180746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.180726 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f338e4/simulator-w1-f338e4-kserve-795c68bcb7-2p9q7"] Nov 01 23:11:51.787616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:51.787561 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:52.303378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.303328 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:52.455598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.454692 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:52.588352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.588325 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 23:11:52.602918 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.602885 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm"] Nov 01 23:11:52.606453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.606428 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-56b833-64hhm"] Nov 01 23:11:52.631689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.631587 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:52.643076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.643053 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 23:11:52.672621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.672592 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-plugins-config-volume\") pod \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " Nov 01 23:11:52.672788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.672654 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ac8ace-f63d-4fb0-b642-5653bde50da7-kube-api-access-8c6jd\" (UniqueName: \"kubernetes.io/projected/81ac8ace-f63d-4fb0-b642-5653bde50da7-kube-api-access-8c6jd\") pod \"81ac8ace-f63d-4fb0-b642-5653bde50da7\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " Nov 01 23:11:52.672788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.672680 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-kube-api-access-d5fml\" (UniqueName: \"kubernetes.io/projected/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-kube-api-access-d5fml\") pod \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\" (UID: \"cdbf0e48-9d53-4450-9584-5969ecc8c1b5\") " Nov 01 23:11:52.673015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.672987 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-plugins-config-volume" pod "cdbf0e48-9d53-4450-9584-5969ecc8c1b5" (UID: "cdbf0e48-9d53-4450-9584-5969ecc8c1b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:52.673175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.673155 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:52.674915 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.674887 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ac8ace-f63d-4fb0-b642-5653bde50da7-kube-api-access-8c6jd" pod "81ac8ace-f63d-4fb0-b642-5653bde50da7" (UID: "81ac8ace-f63d-4fb0-b642-5653bde50da7"). InnerVolumeSpecName "kube-api-access-8c6jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:52.675030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.674995 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-kube-api-access-d5fml" pod "cdbf0e48-9d53-4450-9584-5969ecc8c1b5" (UID: "cdbf0e48-9d53-4450-9584-5969ecc8c1b5"). InnerVolumeSpecName "kube-api-access-d5fml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:11:52.742055 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.742015 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:52.751635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.751604 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ffdcd5-d782-41d5-bbd7-92a219ff64e7" path="/var/lib/kubelet/pods/18ffdcd5-d782-41d5-bbd7-92a219ff64e7/volumes" Nov 01 23:11:52.752094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.752080 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" path="/var/lib/kubelet/pods/a2e8cc95-9482-47a6-b8c2-2507116356d4/volumes" Nov 01 23:11:52.774079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.774053 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ac8ace-f63d-4fb0-b642-5653bde50da7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ac8ace-f63d-4fb0-b642-5653bde50da7-plugins-config-volume\") pod \"81ac8ace-f63d-4fb0-b642-5653bde50da7\" (UID: \"81ac8ace-f63d-4fb0-b642-5653bde50da7\") " Nov 01 23:11:52.774311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.774298 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c6jd\" (UniqueName: \"kubernetes.io/projected/81ac8ace-f63d-4fb0-b642-5653bde50da7-kube-api-access-8c6jd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:52.774358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.774314 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5fml\" (UniqueName: \"kubernetes.io/projected/cdbf0e48-9d53-4450-9584-5969ecc8c1b5-kube-api-access-d5fml\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:52.774393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.774378 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ac8ace-f63d-4fb0-b642-5653bde50da7-plugins-config-volume" pod "81ac8ace-f63d-4fb0-b642-5653bde50da7" (UID: "81ac8ace-f63d-4fb0-b642-5653bde50da7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:11:52.875764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:52.875682 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81ac8ace-f63d-4fb0-b642-5653bde50da7-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:11:53.104616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.104536 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:53.169519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.169425 4981 generic.go:356] "Generic (PLEG): container finished" podID="81ac8ace-f63d-4fb0-b642-5653bde50da7" containerID="f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8" exitCode=0 Nov 01 23:11:53.169519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.169468 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" Nov 01 23:11:53.169519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.169506 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" event={"ID":"81ac8ace-f63d-4fb0-b642-5653bde50da7","Type":"ContainerDied","Data":"f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8"} Nov 01 23:11:53.169836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.169552 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp" event={"ID":"81ac8ace-f63d-4fb0-b642-5653bde50da7","Type":"ContainerDied","Data":"25f8dfdc894c08c876b5e21829e2b6d3633bb05071e3d2a05335971da28822fa"} Nov 01 23:11:53.169836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.169610 4981 scope.go:122] "RemoveContainer" containerID="f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8" Nov 01 23:11:53.171099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.171065 4981 generic.go:356] "Generic (PLEG): container finished" podID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" containerID="171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9" exitCode=0 Nov 01 23:11:53.171222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.171120 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" Nov 01 23:11:53.171222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.171131 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" event={"ID":"cdbf0e48-9d53-4450-9584-5969ecc8c1b5","Type":"ContainerDied","Data":"171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9"} Nov 01 23:11:53.171222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.171196 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh" event={"ID":"cdbf0e48-9d53-4450-9584-5969ecc8c1b5","Type":"ContainerDied","Data":"64d9e61b91f31352c65bf3f2953b81a29833a647f7c4d5c6536f14626f8e6f67"} Nov 01 23:11:53.182346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.182328 4981 scope.go:122] "RemoveContainer" containerID="f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8" Nov 01 23:11:53.182639 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:53.182619 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8\": container with ID starting with f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8 not found: ID does not exist" containerID="f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8" Nov 01 23:11:53.182709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.182647 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8"} err="failed to get container status \"f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8\": rpc error: code = NotFound desc = could not find container \"f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8\": container with ID starting with f6e5554c37bfd508c1cd06a69c9a1405483bfb08e2d489fd38f84eb9f45b0ef8 not found: ID does not exist" Nov 01 23:11:53.182709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.182666 4981 scope.go:122] "RemoveContainer" containerID="171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9" Nov 01 23:11:53.199372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.199351 4981 scope.go:122] "RemoveContainer" containerID="171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9" Nov 01 23:11:53.199709 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:11:53.199686 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9\": container with ID starting with 171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9 not found: ID does not exist" containerID="171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9" Nov 01 23:11:53.199793 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.199717 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9"} err="failed to get container status \"171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9\": rpc error: code = NotFound desc = could not find container \"171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9\": container with ID starting with 171b7c1db5e55168763e421545d8138381d82eeb9b4ee8abb7cfd6561eb830b9 not found: ID does not exist" Nov 01 23:11:53.201234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.201215 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh"] Nov 01 23:11:53.210692 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.210668 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-7366bf-6f55ccc995-qw4mh"] Nov 01 23:11:53.228928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.228905 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp"] Nov 01 23:11:53.232176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.232152 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-7366bf-7559c5b855-n8fzp"] Nov 01 23:11:53.482769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:53.482728 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:54.374772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:54.374732 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:54.746265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:54.746221 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ac8ace-f63d-4fb0-b642-5653bde50da7" path="/var/lib/kubelet/pods/81ac8ace-f63d-4fb0-b642-5653bde50da7/volumes" Nov 01 23:11:54.746871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:54.746855 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" path="/var/lib/kubelet/pods/cdbf0e48-9d53-4450-9584-5969ecc8c1b5/volumes" Nov 01 23:11:54.811502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:54.811462 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd"] Nov 01 23:11:54.829985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:54.829942 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp"] Nov 01 23:11:54.921709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:54.921679 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:55.031192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:55.031103 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:55.274008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:55.273982 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:55.885545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:55.885513 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:56.431833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.431794 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:56.542370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.542334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:56.615717 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.615681 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk"] Nov 01 23:11:56.618870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.618848 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ee07ad-l6bbk"] Nov 01 23:11:56.644730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.644700 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:56.758823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.758707 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:56.761338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.761312 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea539f1-ccaf-4be4-94ce-25bf78db0a2a" path="/var/lib/kubelet/pods/cea539f1-ccaf-4be4-94ce-25bf78db0a2a/volumes" Nov 01 23:11:56.817505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.817464 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp"] Nov 01 23:11:56.817782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.817755 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" containerID="cri-o://eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb" gracePeriod=30 Nov 01 23:11:56.847638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.847562 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:56.882293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.882264 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc"] Nov 01 23:11:56.887895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.887860 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-f338e4-w66kc"] Nov 01 23:11:56.939021 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:56.938985 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:57.029819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.029734 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz"] Nov 01 23:11:57.036188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.036157 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6fe131d-wmjwz"] Nov 01 23:11:57.056170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.056137 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:57.155498 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.155469 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:57.255200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.255167 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:57.358392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.358312 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:57.373696 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.373666 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt"] Nov 01 23:11:57.374224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374211 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" Nov 01 23:11:57.374284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374226 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" Nov 01 23:11:57.374284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374244 4981 cpu_manager.go:414] "containerMap: removing container" podUID="81ac8ace-f63d-4fb0-b642-5653bde50da7" containerName="payload-pre-processing" Nov 01 23:11:57.374284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374250 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ac8ace-f63d-4fb0-b642-5653bde50da7" containerName="payload-pre-processing" Nov 01 23:11:57.374284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374268 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9a487583-0870-4da6-b967-5ec01928d764" containerName="revoke-keys" Nov 01 23:11:57.374284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374273 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a487583-0870-4da6-b967-5ec01928d764" containerName="revoke-keys" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374286 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" containerName="payload-processing" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374292 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" containerName="payload-processing" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374302 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" containerName="revoke-keys" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374308 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" containerName="revoke-keys" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374325 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cea539f1-ccaf-4be4-94ce-25bf78db0a2a" containerName="revoke-keys" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374330 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea539f1-ccaf-4be4-94ce-25bf78db0a2a" containerName="revoke-keys" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374401 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e8cc95-9482-47a6-b8c2-2507116356d4" containerName="main" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374410 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ac8ace-f63d-4fb0-b642-5653bde50da7" containerName="payload-pre-processing" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374419 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a487583-0870-4da6-b967-5ec01928d764" containerName="revoke-keys" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374426 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdbf0e48-9d53-4450-9584-5969ecc8c1b5" containerName="payload-processing" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374432 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" containerName="revoke-keys" Nov 01 23:11:57.374457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.374439 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea539f1-ccaf-4be4-94ce-25bf78db0a2a" containerName="revoke-keys" Nov 01 23:11:57.379436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.379419 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.387380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.387352 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt"] Nov 01 23:11:57.418950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.418918 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:11:57.418950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.418918 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:11:57.436016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.435987 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.436161 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.436147 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.436232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.436187 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm276\" (UniqueName: \"kubernetes.io/projected/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-kube-api-access-wm276\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.442941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.442912 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:11:57.443087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.443072 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:11:57.536906 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.536865 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.537090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.536918 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm276\" (UniqueName: \"kubernetes.io/projected/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-kube-api-access-wm276\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.537090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.536982 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.537687 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.537663 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.539281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.539258 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.544964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.544943 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm276\" (UniqueName: \"kubernetes.io/projected/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-kube-api-access-wm276\") pod \"maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.695949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.695917 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:11:57.976918 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:11:57.976875 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d0b4e8_abed_4565_91b8_f2dde4cea4f9.slice/crio-6d247126b788ce4a4f4bf312f54486f1700882439816da46f97eebaa61b2c089 WatchSource:0}: Error finding container 6d247126b788ce4a4f4bf312f54486f1700882439816da46f97eebaa61b2c089: Status 404 returned error can't find the container with id 6d247126b788ce4a4f4bf312f54486f1700882439816da46f97eebaa61b2c089 Nov 01 23:11:57.977354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:57.977054 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt"] Nov 01 23:11:58.131547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:58.131385 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:58.217360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:58.217324 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" event={"ID":"48d0b4e8-abed-4565-91b8-f2dde4cea4f9","Type":"ContainerStarted","Data":"6d247126b788ce4a4f4bf312f54486f1700882439816da46f97eebaa61b2c089"} Nov 01 23:11:58.735239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:58.735207 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f40cf22-b453-4c6e-89e9-f08d11c413d9" path="/var/lib/kubelet/pods/3f40cf22-b453-4c6e-89e9-f08d11c413d9/volumes" Nov 01 23:11:58.735787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:58.735766 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5affcc59-c738-44ad-a221-9b452f780e8b" path="/var/lib/kubelet/pods/5affcc59-c738-44ad-a221-9b452f780e8b/volumes" Nov 01 23:11:58.970427 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:58.970391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:59.227568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:59.227528 4981 generic.go:356] "Generic (PLEG): container finished" podID="48d0b4e8-abed-4565-91b8-f2dde4cea4f9" containerID="c0b3357551d61159a19984a26bb808a6cc35ffcaf3827f15fd1a01f866a76674" exitCode=0 Nov 01 23:11:59.228011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:59.227618 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" event={"ID":"48d0b4e8-abed-4565-91b8-f2dde4cea4f9","Type":"ContainerDied","Data":"c0b3357551d61159a19984a26bb808a6cc35ffcaf3827f15fd1a01f866a76674"} Nov 01 23:11:59.691586 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:59.691552 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:59.791825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:59.791790 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:11:59.821380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:59.821300 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" podUID="1428b87e-8632-484b-a440-73a44e8aec59" containerName="payload-processing" containerID="cri-o://ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28" gracePeriod=25 Nov 01 23:11:59.833369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:11:59.830277 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerName="payload-pre-processing" containerID="cri-o://02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4" gracePeriod=25 Nov 01 23:12:00.004736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.004651 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:00.395917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.395892 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:12:00.424932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.424897 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:00.474169 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.474139 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-openshift-ca\") pod \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " Nov 01 23:12:00.474345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.474177 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-xks-ca\") pod \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " Nov 01 23:12:00.474345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.474215 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-kube-api-access-wm276\" (UniqueName: \"kubernetes.io/projected/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-kube-api-access-wm276\") pod \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\" (UID: \"48d0b4e8-abed-4565-91b8-f2dde4cea4f9\") " Nov 01 23:12:00.474567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.474542 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-openshift-ca" pod "48d0b4e8-abed-4565-91b8-f2dde4cea4f9" (UID: "48d0b4e8-abed-4565-91b8-f2dde4cea4f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:00.474748 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.474730 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:00.476314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.476290 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-xks-ca" pod "48d0b4e8-abed-4565-91b8-f2dde4cea4f9" (UID: "48d0b4e8-abed-4565-91b8-f2dde4cea4f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:00.476314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.476299 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-kube-api-access-wm276" pod "48d0b4e8-abed-4565-91b8-f2dde4cea4f9" (UID: "48d0b4e8-abed-4565-91b8-f2dde4cea4f9"). InnerVolumeSpecName "kube-api-access-wm276". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:00.514376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.514122 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:00.569922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.569846 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst"] Nov 01 23:12:00.570405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.570391 4981 cpu_manager.go:414] "containerMap: removing container" podUID="48d0b4e8-abed-4565-91b8-f2dde4cea4f9" containerName="revoke-keys" Nov 01 23:12:00.570448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.570406 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d0b4e8-abed-4565-91b8-f2dde4cea4f9" containerName="revoke-keys" Nov 01 23:12:00.570503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.570494 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d0b4e8-abed-4565-91b8-f2dde4cea4f9" containerName="revoke-keys" Nov 01 23:12:00.574099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.574076 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.576170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.576146 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:00.576257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.576173 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm276\" (UniqueName: \"kubernetes.io/projected/48d0b4e8-abed-4565-91b8-f2dde4cea4f9-kube-api-access-wm276\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:00.584517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.584491 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst"] Nov 01 23:12:00.677015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.676977 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.677200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.677042 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.677200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.677155 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsj7b\" (UniqueName: \"kubernetes.io/projected/21edd4d5-f9e2-4943-9801-861a388ecaee-kube-api-access-wsj7b\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.778450 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.778415 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.778627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.778468 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.778627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.778528 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsj7b\" (UniqueName: \"kubernetes.io/projected/21edd4d5-f9e2-4943-9801-861a388ecaee-kube-api-access-wsj7b\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.779209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.779184 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.781034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.781010 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.787358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.787337 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsj7b\" (UniqueName: \"kubernetes.io/projected/21edd4d5-f9e2-4943-9801-861a388ecaee-kube-api-access-wsj7b\") pod \"maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:00.892228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:00.892153 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:01.171011 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:12:01.170983 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21edd4d5_f9e2_4943_9801_861a388ecaee.slice/crio-7d0bc316c2b7e17a75d4bacdfdc139ee1d6ef98b3ddeb919b99fe6cf06bd9bbc WatchSource:0}: Error finding container 7d0bc316c2b7e17a75d4bacdfdc139ee1d6ef98b3ddeb919b99fe6cf06bd9bbc: Status 404 returned error can't find the container with id 7d0bc316c2b7e17a75d4bacdfdc139ee1d6ef98b3ddeb919b99fe6cf06bd9bbc Nov 01 23:12:01.172724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:01.172699 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst"] Nov 01 23:12:01.246285 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:01.246256 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" Nov 01 23:12:01.246438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:01.246253 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt" event={"ID":"48d0b4e8-abed-4565-91b8-f2dde4cea4f9","Type":"ContainerDied","Data":"6d247126b788ce4a4f4bf312f54486f1700882439816da46f97eebaa61b2c089"} Nov 01 23:12:01.246438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:01.246367 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d247126b788ce4a4f4bf312f54486f1700882439816da46f97eebaa61b2c089" Nov 01 23:12:01.247819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:01.247792 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" event={"ID":"21edd4d5-f9e2-4943-9801-861a388ecaee","Type":"ContainerStarted","Data":"7d0bc316c2b7e17a75d4bacdfdc139ee1d6ef98b3ddeb919b99fe6cf06bd9bbc"} Nov 01 23:12:01.409155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:01.409124 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:02.059742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.059677 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:02.146202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.146165 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:12:02.199764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.199727 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4"] Nov 01 23:12:02.205019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.204972 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-envoy\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205093 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-data\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205171 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-credential-socket\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205203 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-certs\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205288 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-socket\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205342 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-podinfo\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205383 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istiod-ca-cert\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205424 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-kube-api-access-ttfg5\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-kube-api-access-ttfg5\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205754 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205548 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-token\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-token\") pod \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\" (UID: \"b67106f5-6491-4f06-a60e-c5cf7d0d6c64\") " Nov 01 23:12:02.205754 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205618 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-credential-socket" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.205754 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205658 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-data" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.205985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.205944 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-certs" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.206144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.206122 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.206224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.206148 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.206224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.206164 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.206505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.206481 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istiod-ca-cert" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:02.206983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.206816 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-socket" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.209311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.209288 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-kube-api-access-ttfg5" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "kube-api-access-ttfg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.209474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.209444 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-podinfo" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:02.211398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.211367 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-envoy" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:02.212696 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.212673 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-token" pod "b67106f5-6491-4f06-a60e-c5cf7d0d6c64" (UID: "b67106f5-6491-4f06-a60e-c5cf7d0d6c64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:02.213245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.213220 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc"] Nov 01 23:12:02.228663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.228635 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z"] Nov 01 23:12:02.257970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.257936 4981 generic.go:356] "Generic (PLEG): container finished" podID="21edd4d5-f9e2-4943-9801-861a388ecaee" containerID="18b6b9c2637b5300a26a94d61017b690c2c9140e5caf19a7ee9f6435c2737768" exitCode=0 Nov 01 23:12:02.258149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.258017 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" event={"ID":"21edd4d5-f9e2-4943-9801-861a388ecaee","Type":"ContainerDied","Data":"18b6b9c2637b5300a26a94d61017b690c2c9140e5caf19a7ee9f6435c2737768"} Nov 01 23:12:02.259712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.259681 4981 generic.go:356] "Generic (PLEG): container finished" podID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerID="eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb" exitCode=0 Nov 01 23:12:02.259936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.259915 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" event={"ID":"b67106f5-6491-4f06-a60e-c5cf7d0d6c64","Type":"ContainerDied","Data":"eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb"} Nov 01 23:12:02.260046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.259959 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" event={"ID":"b67106f5-6491-4f06-a60e-c5cf7d0d6c64","Type":"ContainerDied","Data":"174a301a84f8336319eb98075b824d40abde5e13bb4efe09a862ca6246977ebd"} Nov 01 23:12:02.260046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.259982 4981 scope.go:122] "RemoveContainer" containerID="eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb" Nov 01 23:12:02.260155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.260117 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp" Nov 01 23:12:02.273831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.273798 4981 scope.go:122] "RemoveContainer" containerID="eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb" Nov 01 23:12:02.274188 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:02.274162 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb\": container with ID starting with eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb not found: ID does not exist" containerID="eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb" Nov 01 23:12:02.274283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.274195 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb"} err="failed to get container status \"eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb\": rpc error: code = NotFound desc = could not find container \"eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb\": container with ID starting with eecb7453b7eceea14e611aee01e36594fd5a4aecb5e06a260c6d59d0a86d8ccb not found: ID does not exist" Nov 01 23:12:02.299495 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.299466 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:02.307480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.307448 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp"] Nov 01 23:12:02.307820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.307799 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.307911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.307823 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.307911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.307834 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.307911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.307844 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.307911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.307852 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.307911 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.307861 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttfg5\" (UniqueName: \"kubernetes.io/projected/b67106f5-6491-4f06-a60e-c5cf7d0d6c64-kube-api-access-ttfg5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:02.313847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.313813 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-075771-openshift-default-5fbd95bc45-sppgp"] Nov 01 23:12:02.735771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.735739 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" path="/var/lib/kubelet/pods/b67106f5-6491-4f06-a60e-c5cf7d0d6c64/volumes" Nov 01 23:12:02.990455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:02.990371 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:03.152213 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.152175 4981 scope.go:122] "RemoveContainer" containerID="650983d678dd5a39ace98eb587b45799138b31b9cb9c2cae78f3203e68904f4b" Nov 01 23:12:03.163075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.163051 4981 scope.go:122] "RemoveContainer" containerID="9e73464e555ee859071b35cb510d62acb5d5903f50cb130bb6212a5948df69d6" Nov 01 23:12:03.465718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.465691 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:03.522750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.522668 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-openshift-ca\") pod \"21edd4d5-f9e2-4943-9801-861a388ecaee\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " Nov 01 23:12:03.522939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.522768 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21edd4d5-f9e2-4943-9801-861a388ecaee-kube-api-access-wsj7b\" (UniqueName: \"kubernetes.io/projected/21edd4d5-f9e2-4943-9801-861a388ecaee-kube-api-access-wsj7b\") pod \"21edd4d5-f9e2-4943-9801-861a388ecaee\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " Nov 01 23:12:03.522939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.522891 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-xks-ca\") pod \"21edd4d5-f9e2-4943-9801-861a388ecaee\" (UID: \"21edd4d5-f9e2-4943-9801-861a388ecaee\") " Nov 01 23:12:03.523137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.523106 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-openshift-ca" pod "21edd4d5-f9e2-4943-9801-861a388ecaee" (UID: "21edd4d5-f9e2-4943-9801-861a388ecaee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:03.523329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.523303 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:03.524992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.524966 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21edd4d5-f9e2-4943-9801-861a388ecaee-kube-api-access-wsj7b" pod "21edd4d5-f9e2-4943-9801-861a388ecaee" (UID: "21edd4d5-f9e2-4943-9801-861a388ecaee"). InnerVolumeSpecName "kube-api-access-wsj7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:03.525386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.525361 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-xks-ca" pod "21edd4d5-f9e2-4943-9801-861a388ecaee" (UID: "21edd4d5-f9e2-4943-9801-861a388ecaee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:03.624515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.624476 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsj7b\" (UniqueName: \"kubernetes.io/projected/21edd4d5-f9e2-4943-9801-861a388ecaee-kube-api-access-wsj7b\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:03.624515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:03.624509 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21edd4d5-f9e2-4943-9801-861a388ecaee-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:04.082442 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:04.082408 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:04.280139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:04.280103 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" Nov 01 23:12:04.280424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:04.280134 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst" event={"ID":"21edd4d5-f9e2-4943-9801-861a388ecaee","Type":"ContainerDied","Data":"7d0bc316c2b7e17a75d4bacdfdc139ee1d6ef98b3ddeb919b99fe6cf06bd9bbc"} Nov 01 23:12:04.280424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:04.280185 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d0bc316c2b7e17a75d4bacdfdc139ee1d6ef98b3ddeb919b99fe6cf06bd9bbc" Nov 01 23:12:04.967859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:04.967743 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:05.488079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.488037 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:05.492097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.492074 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn"] Nov 01 23:12:05.517627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.517595 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx"] Nov 01 23:12:05.603656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.603625 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:05.727858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.727750 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:05.754813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.754728 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4"] Nov 01 23:12:05.754999 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.754940 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" containerID="cri-o://3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d" gracePeriod=30 Nov 01 23:12:05.833621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.833532 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:05.987027 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.986990 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4"] Nov 01 23:12:05.987261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:05.987238 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" containerID="cri-o://2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e" gracePeriod=30 Nov 01 23:12:06.014309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.014226 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:06.177355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.177321 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj"] Nov 01 23:12:06.177900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.177886 4981 cpu_manager.go:414] "containerMap: removing container" podUID="21edd4d5-f9e2-4943-9801-861a388ecaee" containerName="revoke-keys" Nov 01 23:12:06.177958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.177902 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21edd4d5-f9e2-4943-9801-861a388ecaee" containerName="revoke-keys" Nov 01 23:12:06.177997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.177991 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" Nov 01 23:12:06.178036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.177999 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" Nov 01 23:12:06.178105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.178096 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="21edd4d5-f9e2-4943-9801-861a388ecaee" containerName="revoke-keys" Nov 01 23:12:06.178143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.178114 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67106f5-6491-4f06-a60e-c5cf7d0d6c64" containerName="istio-proxy" Nov 01 23:12:06.181897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.181879 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.187267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.187241 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:12:06.187267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.187260 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:12:06.212593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.191239 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj"] Nov 01 23:12:06.261798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.261762 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.262009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.261832 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtkq\" (UniqueName: \"kubernetes.io/projected/a3566773-6869-412b-ac8a-cfe1cd9c3194-kube-api-access-hmtkq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.262086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.262026 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.278440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.278377 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:12:06.278653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.278640 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:12:06.363484 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.363444 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.363728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.363567 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.363728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.363668 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtkq\" (UniqueName: \"kubernetes.io/projected/a3566773-6869-412b-ac8a-cfe1cd9c3194-kube-api-access-hmtkq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.364185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.364160 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.365858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.365830 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.372488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.372465 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtkq\" (UniqueName: \"kubernetes.io/projected/a3566773-6869-412b-ac8a-cfe1cd9c3194-kube-api-access-hmtkq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.531493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.531408 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:06.749708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.749677 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:06.817955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:06.817923 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj"] Nov 01 23:12:06.818466 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:12:06.818442 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3566773_6869_412b_ac8a_cfe1cd9c3194.slice/crio-7eef25f04bdacc00c1ae36e7228ca7849ddf9987ba71393bb09c27dacec026e7 WatchSource:0}: Error finding container 7eef25f04bdacc00c1ae36e7228ca7849ddf9987ba71393bb09c27dacec026e7: Status 404 returned error can't find the container with id 7eef25f04bdacc00c1ae36e7228ca7849ddf9987ba71393bb09c27dacec026e7 Nov 01 23:12:07.304897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.304859 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" event={"ID":"a3566773-6869-412b-ac8a-cfe1cd9c3194","Type":"ContainerStarted","Data":"1301923a48f4adfe25f7595662b83353a4b706f251d69c7eb74a4018d56b0ffc"} Nov 01 23:12:07.305097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.304928 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" event={"ID":"a3566773-6869-412b-ac8a-cfe1cd9c3194","Type":"ContainerStarted","Data":"7eef25f04bdacc00c1ae36e7228ca7849ddf9987ba71393bb09c27dacec026e7"} Nov 01 23:12:07.323210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.323165 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" podStartSLOduration=1.323151754 podStartE2EDuration="1.323151754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:07.322018736 +0000 UTC m=+2892.749896672" watchObservedRunningTime="2026-11-01 23:12:07.323151754 +0000 UTC m=+2892.751029684" Nov 01 23:12:07.491102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.491061 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" podUID="1428b87e-8632-484b-a440-73a44e8aec59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:07.491102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.491097 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:07.507129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.507101 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c"] Nov 01 23:12:07.526555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.526525 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk"] Nov 01 23:12:07.586833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.586799 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:07.687223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:07.687192 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:08.022221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:08.022190 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn"] Nov 01 23:12:08.025625 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:08.025594 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c322b61-kfgcn"] Nov 01 23:12:08.255446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:08.255411 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:08.326384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:08.326287 4981 generic.go:356] "Generic (PLEG): container finished" podID="a3566773-6869-412b-ac8a-cfe1cd9c3194" containerID="1301923a48f4adfe25f7595662b83353a4b706f251d69c7eb74a4018d56b0ffc" exitCode=0 Nov 01 23:12:08.326558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:08.326376 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" event={"ID":"a3566773-6869-412b-ac8a-cfe1cd9c3194","Type":"ContainerDied","Data":"1301923a48f4adfe25f7595662b83353a4b706f251d69c7eb74a4018d56b0ffc"} Nov 01 23:12:08.735763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:08.735727 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bed368-31e8-4ab9-aaee-cc8b501d0e9e" path="/var/lib/kubelet/pods/12bed368-31e8-4ab9-aaee-cc8b501d0e9e/volumes" Nov 01 23:12:08.857728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:08.857662 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:09.355255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.355221 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876"] Nov 01 23:12:09.359509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.359491 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.369777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.369745 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876"] Nov 01 23:12:09.496428 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.496404 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:09.510354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.510330 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.510453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.510376 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nchl\" (UniqueName: \"kubernetes.io/projected/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-kube-api-access-7nchl\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.510543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.510522 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.612152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612066 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-openshift-ca\") pod \"a3566773-6869-412b-ac8a-cfe1cd9c3194\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " Nov 01 23:12:09.612152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612111 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3566773-6869-412b-ac8a-cfe1cd9c3194-kube-api-access-hmtkq\" (UniqueName: \"kubernetes.io/projected/a3566773-6869-412b-ac8a-cfe1cd9c3194-kube-api-access-hmtkq\") pod \"a3566773-6869-412b-ac8a-cfe1cd9c3194\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " Nov 01 23:12:09.612393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612168 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-xks-ca\") pod \"a3566773-6869-412b-ac8a-cfe1cd9c3194\" (UID: \"a3566773-6869-412b-ac8a-cfe1cd9c3194\") " Nov 01 23:12:09.612455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612413 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-openshift-ca" pod "a3566773-6869-412b-ac8a-cfe1cd9c3194" (UID: "a3566773-6869-412b-ac8a-cfe1cd9c3194"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:09.612496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612483 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.612551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612536 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nchl\" (UniqueName: \"kubernetes.io/projected/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-kube-api-access-7nchl\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.612695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612673 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.612798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.612780 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:09.613512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.613481 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.614453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.614421 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3566773-6869-412b-ac8a-cfe1cd9c3194-kube-api-access-hmtkq" pod "a3566773-6869-412b-ac8a-cfe1cd9c3194" (UID: "a3566773-6869-412b-ac8a-cfe1cd9c3194"). InnerVolumeSpecName "kube-api-access-hmtkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:09.614941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.614918 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.614941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.614928 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-xks-ca" pod "a3566773-6869-412b-ac8a-cfe1cd9c3194" (UID: "a3566773-6869-412b-ac8a-cfe1cd9c3194"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:09.624191 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.624136 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nchl\" (UniqueName: \"kubernetes.io/projected/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-kube-api-access-7nchl\") pod \"maas-api-revoke-sub-1d32c89d-bw876\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.646375 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.646347 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:09.677511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.677485 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:09.714365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.714335 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmtkq\" (UniqueName: \"kubernetes.io/projected/a3566773-6869-412b-ac8a-cfe1cd9c3194-kube-api-access-hmtkq\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:09.714365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.714367 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3566773-6869-412b-ac8a-cfe1cd9c3194-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:09.986711 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:09.986687 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876"] Nov 01 23:12:09.987198 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:12:09.987171 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0911c3c_3b05_4f9b_8de7_5f21dd85c707.slice/crio-b864237984152706144701e70ea405216bae8dd7cbf532141b64a388c6644753 WatchSource:0}: Error finding container b864237984152706144701e70ea405216bae8dd7cbf532141b64a388c6644753: Status 404 returned error can't find the container with id b864237984152706144701e70ea405216bae8dd7cbf532141b64a388c6644753 Nov 01 23:12:10.342067 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.342035 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" Nov 01 23:12:10.342067 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.342050 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj" event={"ID":"a3566773-6869-412b-ac8a-cfe1cd9c3194","Type":"ContainerDied","Data":"7eef25f04bdacc00c1ae36e7228ca7849ddf9987ba71393bb09c27dacec026e7"} Nov 01 23:12:10.342336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.342097 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eef25f04bdacc00c1ae36e7228ca7849ddf9987ba71393bb09c27dacec026e7" Nov 01 23:12:10.343848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.343823 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" event={"ID":"c0911c3c-3b05-4f9b-8de7-5f21dd85c707","Type":"ContainerStarted","Data":"2ca857f423cd9e4d7f5509c8cecbf5638c3e08bdc9fd433535c471a96ffbb8db"} Nov 01 23:12:10.343971 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.343872 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" event={"ID":"c0911c3c-3b05-4f9b-8de7-5f21dd85c707","Type":"ContainerStarted","Data":"b864237984152706144701e70ea405216bae8dd7cbf532141b64a388c6644753"} Nov 01 23:12:10.369977 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.369946 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:10.370221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.370165 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" podStartSLOduration=1.370153469 podStartE2EDuration="1.370153469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:10.367923046 +0000 UTC m=+2895.795800975" watchObservedRunningTime="2026-11-01 23:12:10.370153469 +0000 UTC m=+2895.798031399" Nov 01 23:12:10.689530 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.689493 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c"] Nov 01 23:12:10.764246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.764205 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:10.785780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.785744 4981 patch_prober.go:29] interesting pod/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:12:10.785966 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.785810 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:12:10.864398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:10.864363 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:11.065903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.065871 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:11.110396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.110371 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:12:11.237785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.237704 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-envoy\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.237785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.237742 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7885a556-eb5d-479e-9744-7607d03fc3af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7885a556-eb5d-479e-9744-7607d03fc3af-istio-podinfo\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.237785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.237771 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-certs\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.238092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.237808 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-socket\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.238092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.237841 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7885a556-eb5d-479e-9744-7607d03fc3af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7885a556-eb5d-479e-9744-7607d03fc3af-istiod-ca-cert\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.238092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.237868 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-data\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.238092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238012 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-istio-token\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-istio-token\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.238345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238118 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-credential-socket\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.238345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238145 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-socket" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.238345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238153 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-kube-api-access-mf77v\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-kube-api-access-mf77v\") pod \"7885a556-eb5d-479e-9744-7607d03fc3af\" (UID: \"7885a556-eb5d-479e-9744-7607d03fc3af\") " Nov 01 23:12:11.238345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238259 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7885a556-eb5d-479e-9744-7607d03fc3af-istiod-ca-cert" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:11.238612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238349 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-credential-socket" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.238612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238524 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-data" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.238756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238727 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.238821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238758 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.238821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238778 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7885a556-eb5d-479e-9744-7607d03fc3af-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.238821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238799 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.238821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.238727 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-certs" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.240264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.240243 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-kube-api-access-mf77v" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "kube-api-access-mf77v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.240460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.240432 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-envoy" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.240460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.240456 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-istio-token" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.240670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.240506 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7885a556-eb5d-479e-9744-7607d03fc3af-istio-podinfo" pod "7885a556-eb5d-479e-9744-7607d03fc3af" (UID: "7885a556-eb5d-479e-9744-7607d03fc3af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:11.258515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.258494 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 23:12:11.339249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339175 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-envoy\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339316 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istiod-ca-cert\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339344 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-podinfo\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339387 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-certs\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339405 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-credential-socket\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339464 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-token\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-token\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339503 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-socket\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339550 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-kube-api-access-rvzkm\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-kube-api-access-rvzkm\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.339707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339594 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-data\") pod \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\" (UID: \"5d11122e-79a5-4d13-a813-3d4c64ab1a9c\") " Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339733 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-certs" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339856 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-socket" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339878 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istiod-ca-cert" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.339988 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-credential-socket" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340031 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-data" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340225 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340248 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340269 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340286 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340300 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf77v\" (UniqueName: \"kubernetes.io/projected/7885a556-eb5d-479e-9744-7607d03fc3af-kube-api-access-mf77v\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340315 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340328 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340344 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7885a556-eb5d-479e-9744-7607d03fc3af-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340356 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.340649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.340365 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7885a556-eb5d-479e-9744-7607d03fc3af-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.341915 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.341888 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-envoy" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:11.342032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.341965 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-podinfo" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:11.342108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.342087 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-kube-api-access-rvzkm" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "kube-api-access-rvzkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.342163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.342118 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-token" pod "5d11122e-79a5-4d13-a813-3d4c64ab1a9c" (UID: "5d11122e-79a5-4d13-a813-3d4c64ab1a9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:11.352517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.352495 4981 generic.go:356] "Generic (PLEG): container finished" podID="7885a556-eb5d-479e-9744-7607d03fc3af" containerID="3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d" exitCode=0 Nov 01 23:12:11.352655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.352549 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" Nov 01 23:12:11.352655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.352595 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" event={"ID":"7885a556-eb5d-479e-9744-7607d03fc3af","Type":"ContainerDied","Data":"3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d"} Nov 01 23:12:11.352779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.352657 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4" event={"ID":"7885a556-eb5d-479e-9744-7607d03fc3af","Type":"ContainerDied","Data":"276a94711219e34da664d84e453ec8e8a0795cb7d6c53e2ebb1917cf5e2874bd"} Nov 01 23:12:11.352779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.352682 4981 scope.go:122] "RemoveContainer" containerID="3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d" Nov 01 23:12:11.354516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.354497 4981 generic.go:356] "Generic (PLEG): container finished" podID="c0911c3c-3b05-4f9b-8de7-5f21dd85c707" containerID="2ca857f423cd9e4d7f5509c8cecbf5638c3e08bdc9fd433535c471a96ffbb8db" exitCode=0 Nov 01 23:12:11.354664 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.354613 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" event={"ID":"c0911c3c-3b05-4f9b-8de7-5f21dd85c707","Type":"ContainerDied","Data":"2ca857f423cd9e4d7f5509c8cecbf5638c3e08bdc9fd433535c471a96ffbb8db"} Nov 01 23:12:11.356175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.356154 4981 generic.go:356] "Generic (PLEG): container finished" podID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerID="2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e" exitCode=0 Nov 01 23:12:11.356307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.356177 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" event={"ID":"5d11122e-79a5-4d13-a813-3d4c64ab1a9c","Type":"ContainerDied","Data":"2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e"} Nov 01 23:12:11.356307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.356194 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" Nov 01 23:12:11.356307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.356226 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4" event={"ID":"5d11122e-79a5-4d13-a813-3d4c64ab1a9c","Type":"ContainerDied","Data":"f031d1b7edca05bcb0813890e6d8bd587289da287b57491e5c4ec7b583337b18"} Nov 01 23:12:11.367247 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.367219 4981 scope.go:122] "RemoveContainer" containerID="3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d" Nov 01 23:12:11.371226 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:11.371159 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d\": container with ID starting with 3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d not found: ID does not exist" containerID="3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d" Nov 01 23:12:11.371379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.371235 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d"} err="failed to get container status \"3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d\": rpc error: code = NotFound desc = could not find container \"3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d\": container with ID starting with 3b16d15236c32e84e389394287d12af77fb9be12294ec41a69650f7f1b8bc43d not found: ID does not exist" Nov 01 23:12:11.371379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.371304 4981 scope.go:122] "RemoveContainer" containerID="2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e" Nov 01 23:12:11.382979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.382957 4981 scope.go:122] "RemoveContainer" containerID="2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e" Nov 01 23:12:11.383299 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:11.383276 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e\": container with ID starting with 2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e not found: ID does not exist" containerID="2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e" Nov 01 23:12:11.383361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.383308 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e"} err="failed to get container status \"2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e\": rpc error: code = NotFound desc = could not find container \"2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e\": container with ID starting with 2ae00b50fae22cf5b6ef955e76ef91b993ac6f9b3ac00ac7178180da2882713e not found: ID does not exist" Nov 01 23:12:11.394788 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.394756 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4"] Nov 01 23:12:11.398245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.398224 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f338e4-openshift-default-5cc97f9f85-7mtd4"] Nov 01 23:12:11.407508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.407483 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4"] Nov 01 23:12:11.410380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.410358 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ee07ad-openshift-default-7df6bb4695-khsb4"] Nov 01 23:12:11.441292 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.441269 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.441384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.441293 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzkm\" (UniqueName: \"kubernetes.io/projected/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-kube-api-access-rvzkm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.441384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.441304 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.441384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.441312 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d11122e-79a5-4d13-a813-3d4c64ab1a9c-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:11.563886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.563812 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw"] Nov 01 23:12:11.564378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564365 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a3566773-6869-412b-ac8a-cfe1cd9c3194" containerName="revoke-keys" Nov 01 23:12:11.564425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564380 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3566773-6869-412b-ac8a-cfe1cd9c3194" containerName="revoke-keys" Nov 01 23:12:11.564425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564419 4981 cpu_manager.go:414] "containerMap: removing container" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" Nov 01 23:12:11.564425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564425 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" Nov 01 23:12:11.564536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564432 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" Nov 01 23:12:11.564536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564437 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" Nov 01 23:12:11.564536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564507 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" containerName="istio-proxy" Nov 01 23:12:11.564536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564514 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" containerName="istio-proxy" Nov 01 23:12:11.564536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.564527 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3566773-6869-412b-ac8a-cfe1cd9c3194" containerName="revoke-keys" Nov 01 23:12:11.569661 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.569630 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.573198 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.573177 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg"] Nov 01 23:12:11.577548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.577526 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw"] Nov 01 23:12:11.577683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.577620 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.585133 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.585095 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg"] Nov 01 23:12:11.745205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.745167 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.745389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.745295 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.745389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.745351 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.745389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.745369 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.745530 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.745511 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fph5t\" (UniqueName: \"kubernetes.io/projected/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-kube-api-access-fph5t\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.745584 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.745563 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74b5\" (UniqueName: \"kubernetes.io/projected/741ceb40-f226-49d0-b06a-2d1acf59a08f-kube-api-access-l74b5\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.847405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.847308 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.847405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.847392 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.847680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.847474 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.847680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.847501 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.847680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.847562 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fph5t\" (UniqueName: \"kubernetes.io/projected/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-kube-api-access-fph5t\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.847680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.847625 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74b5\" (UniqueName: \"kubernetes.io/projected/741ceb40-f226-49d0-b06a-2d1acf59a08f-kube-api-access-l74b5\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.848237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.848218 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.848421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.848390 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.850264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.850239 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.850782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.850759 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.851527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.851508 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:11.858750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.858727 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74b5\" (UniqueName: \"kubernetes.io/projected/741ceb40-f226-49d0-b06a-2d1acf59a08f-kube-api-access-l74b5\") pod \"maas-api-revoke-sub-7293b400-q75xw\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.858837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.858804 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fph5t\" (UniqueName: \"kubernetes.io/projected/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-kube-api-access-fph5t\") pod \"maas-api-revoke-sub-ded41bf4-4x5kg\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:11.890666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.890640 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:11.898784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:11.898751 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:12.189604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.189556 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw"] Nov 01 23:12:12.189988 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:12:12.189856 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741ceb40_f226_49d0_b06a_2d1acf59a08f.slice/crio-e01ba59b69acc64190cda0c035757c1d7cd3c8fea3353bd9fdd8a7b930ed15a9 WatchSource:0}: Error finding container e01ba59b69acc64190cda0c035757c1d7cd3c8fea3353bd9fdd8a7b930ed15a9: Status 404 returned error can't find the container with id e01ba59b69acc64190cda0c035757c1d7cd3c8fea3353bd9fdd8a7b930ed15a9 Nov 01 23:12:12.191562 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:12:12.191540 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d1d9b2_08ca_4827_a6ff_b35966fc95ba.slice/crio-939090dcad8112826fa4ae61326c04c8520eafcdc5db7ac99cec7d044de7a330 WatchSource:0}: Error finding container 939090dcad8112826fa4ae61326c04c8520eafcdc5db7ac99cec7d044de7a330: Status 404 returned error can't find the container with id 939090dcad8112826fa4ae61326c04c8520eafcdc5db7ac99cec7d044de7a330 Nov 01 23:12:12.193430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.193411 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg"] Nov 01 23:12:12.365820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.365789 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" event={"ID":"38d1d9b2-08ca-4827-a6ff-b35966fc95ba","Type":"ContainerStarted","Data":"939090dcad8112826fa4ae61326c04c8520eafcdc5db7ac99cec7d044de7a330"} Nov 01 23:12:12.367022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.366988 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" event={"ID":"741ceb40-f226-49d0-b06a-2d1acf59a08f","Type":"ContainerStarted","Data":"e01ba59b69acc64190cda0c035757c1d7cd3c8fea3353bd9fdd8a7b930ed15a9"} Nov 01 23:12:12.507449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.507420 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" podUID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" containerName="payload-processing" containerID="cri-o://056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657" gracePeriod=25 Nov 01 23:12:12.514522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.514498 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:12.526900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.526839 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" podUID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" containerName="payload-pre-processing" containerID="cri-o://4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5" gracePeriod=25 Nov 01 23:12:12.590539 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.590371 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:12.659004 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.658973 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-xks-ca\") pod \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " Nov 01 23:12:12.659224 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.659091 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-kube-api-access-7nchl\" (UniqueName: \"kubernetes.io/projected/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-kube-api-access-7nchl\") pod \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " Nov 01 23:12:12.659313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.659236 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-openshift-ca\") pod \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\" (UID: \"c0911c3c-3b05-4f9b-8de7-5f21dd85c707\") " Nov 01 23:12:12.659722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.659693 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-openshift-ca" pod "c0911c3c-3b05-4f9b-8de7-5f21dd85c707" (UID: "c0911c3c-3b05-4f9b-8de7-5f21dd85c707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:12.661703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.661667 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-xks-ca" pod "c0911c3c-3b05-4f9b-8de7-5f21dd85c707" (UID: "c0911c3c-3b05-4f9b-8de7-5f21dd85c707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:12.661809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.661721 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-kube-api-access-7nchl" pod "c0911c3c-3b05-4f9b-8de7-5f21dd85c707" (UID: "c0911c3c-3b05-4f9b-8de7-5f21dd85c707"). InnerVolumeSpecName "kube-api-access-7nchl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:12.688786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.688762 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:12.759783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.758520 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d11122e-79a5-4d13-a813-3d4c64ab1a9c" path="/var/lib/kubelet/pods/5d11122e-79a5-4d13-a813-3d4c64ab1a9c/volumes" Nov 01 23:12:12.759783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.759376 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7885a556-eb5d-479e-9744-7607d03fc3af" path="/var/lib/kubelet/pods/7885a556-eb5d-479e-9744-7607d03fc3af/volumes" Nov 01 23:12:12.762658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.762628 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:12.762658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.762660 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:12.762868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.762677 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nchl\" (UniqueName: \"kubernetes.io/projected/c0911c3c-3b05-4f9b-8de7-5f21dd85c707-kube-api-access-7nchl\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:12.791119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:12.791093 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:13.377940 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.377909 4981 generic.go:356] "Generic (PLEG): container finished" podID="38d1d9b2-08ca-4827-a6ff-b35966fc95ba" containerID="a86eb3dabd0188c921adb7e307968540967aefffc7bf6b1699df93a9e69fdac7" exitCode=0 Nov 01 23:12:13.378369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.377994 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" event={"ID":"38d1d9b2-08ca-4827-a6ff-b35966fc95ba","Type":"ContainerDied","Data":"a86eb3dabd0188c921adb7e307968540967aefffc7bf6b1699df93a9e69fdac7"} Nov 01 23:12:13.379760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.379735 4981 generic.go:356] "Generic (PLEG): container finished" podID="741ceb40-f226-49d0-b06a-2d1acf59a08f" containerID="2e8d39f5e10c5cd4e03e7db875042b4d436bbb6ad7a93a373bc692ca7958ef46" exitCode=0 Nov 01 23:12:13.379853 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.379817 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" event={"ID":"741ceb40-f226-49d0-b06a-2d1acf59a08f","Type":"ContainerDied","Data":"2e8d39f5e10c5cd4e03e7db875042b4d436bbb6ad7a93a373bc692ca7958ef46"} Nov 01 23:12:13.381173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.381158 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" Nov 01 23:12:13.381269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.381196 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876" event={"ID":"c0911c3c-3b05-4f9b-8de7-5f21dd85c707","Type":"ContainerDied","Data":"b864237984152706144701e70ea405216bae8dd7cbf532141b64a388c6644753"} Nov 01 23:12:13.381269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.381237 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b864237984152706144701e70ea405216bae8dd7cbf532141b64a388c6644753" Nov 01 23:12:13.541183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.541157 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:13.951542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:13.951511 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:14.356816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.356732 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:14.540919 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:14.538465 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:12:14.561488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.561462 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:14.591609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.591562 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:14.684637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.684604 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-openshift-ca\") pod \"741ceb40-f226-49d0-b06a-2d1acf59a08f\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " Nov 01 23:12:14.684840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.684673 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-xks-ca\") pod \"741ceb40-f226-49d0-b06a-2d1acf59a08f\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " Nov 01 23:12:14.684840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.684715 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741ceb40-f226-49d0-b06a-2d1acf59a08f-kube-api-access-l74b5\" (UniqueName: \"kubernetes.io/projected/741ceb40-f226-49d0-b06a-2d1acf59a08f-kube-api-access-l74b5\") pod \"741ceb40-f226-49d0-b06a-2d1acf59a08f\" (UID: \"741ceb40-f226-49d0-b06a-2d1acf59a08f\") " Nov 01 23:12:14.684965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.684919 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-openshift-ca" pod "741ceb40-f226-49d0-b06a-2d1acf59a08f" (UID: "741ceb40-f226-49d0-b06a-2d1acf59a08f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:14.685250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.685228 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:14.686820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.686795 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-xks-ca" pod "741ceb40-f226-49d0-b06a-2d1acf59a08f" (UID: "741ceb40-f226-49d0-b06a-2d1acf59a08f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:14.686948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.686930 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741ceb40-f226-49d0-b06a-2d1acf59a08f-kube-api-access-l74b5" pod "741ceb40-f226-49d0-b06a-2d1acf59a08f" (UID: "741ceb40-f226-49d0-b06a-2d1acf59a08f"). InnerVolumeSpecName "kube-api-access-l74b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:14.785813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.785780 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-kube-api-access-fph5t\" (UniqueName: \"kubernetes.io/projected/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-kube-api-access-fph5t\") pod \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " Nov 01 23:12:14.786008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.785854 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-openshift-ca\") pod \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " Nov 01 23:12:14.786008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.785906 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-xks-ca\") pod \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\" (UID: \"38d1d9b2-08ca-4827-a6ff-b35966fc95ba\") " Nov 01 23:12:14.786194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.786171 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-openshift-ca" pod "38d1d9b2-08ca-4827-a6ff-b35966fc95ba" (UID: "38d1d9b2-08ca-4827-a6ff-b35966fc95ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:14.786336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.786323 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:14.786391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.786338 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74b5\" (UniqueName: \"kubernetes.io/projected/741ceb40-f226-49d0-b06a-2d1acf59a08f-kube-api-access-l74b5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:14.786391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.786349 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741ceb40-f226-49d0-b06a-2d1acf59a08f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:14.787832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.787811 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-kube-api-access-fph5t" pod "38d1d9b2-08ca-4827-a6ff-b35966fc95ba" (UID: "38d1d9b2-08ca-4827-a6ff-b35966fc95ba"). InnerVolumeSpecName "kube-api-access-fph5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:14.787900 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.787870 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-xks-ca" pod "38d1d9b2-08ca-4827-a6ff-b35966fc95ba" (UID: "38d1d9b2-08ca-4827-a6ff-b35966fc95ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:14.887780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.887742 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fph5t\" (UniqueName: \"kubernetes.io/projected/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-kube-api-access-fph5t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:14.887780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:14.887776 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d1d9b2-08ca-4827-a6ff-b35966fc95ba-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:15.171499 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.171423 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:15.398485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.398453 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" Nov 01 23:12:15.398694 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.398453 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg" event={"ID":"38d1d9b2-08ca-4827-a6ff-b35966fc95ba","Type":"ContainerDied","Data":"939090dcad8112826fa4ae61326c04c8520eafcdc5db7ac99cec7d044de7a330"} Nov 01 23:12:15.398694 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.398628 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="939090dcad8112826fa4ae61326c04c8520eafcdc5db7ac99cec7d044de7a330" Nov 01 23:12:15.399987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.399964 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" Nov 01 23:12:15.400118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.399994 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw" event={"ID":"741ceb40-f226-49d0-b06a-2d1acf59a08f","Type":"ContainerDied","Data":"e01ba59b69acc64190cda0c035757c1d7cd3c8fea3353bd9fdd8a7b930ed15a9"} Nov 01 23:12:15.400118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.400033 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e01ba59b69acc64190cda0c035757c1d7cd3c8fea3353bd9fdd8a7b930ed15a9" Nov 01 23:12:15.571335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.571306 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:15.690570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.690532 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" podUID="d815af03-1261-4769-bf33-6a76cf1f17bd" containerName="payload-processing" containerID="cri-o://4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef" gracePeriod=25 Nov 01 23:12:15.959956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:15.959929 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:16.211507 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:16.211415 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:16.284713 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:16.284680 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5"] Nov 01 23:12:16.295999 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:16.295965 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd"] Nov 01 23:12:16.363348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:16.363319 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:16.463511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:16.463441 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:16.670316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:16.670271 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:17.229631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.229559 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" containerID="cri-o://2bb0195e003c7f78c178520adb0382ee893bd022ae3d9ff01f0f53bd1517e665" gracePeriod=45 Nov 01 23:12:17.241710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.241668 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" containerID="cri-o://6871eb6391bd584232989b9ded063e04fae5b807df1e2f180155fe55e88c61e6" gracePeriod=45 Nov 01 23:12:17.255000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.254973 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" containerID="cri-o://a61d71a69d8bb4581bf90789c8a4771dc1a189deebe192f76389c972902899a8" gracePeriod=45 Nov 01 23:12:17.296396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.296357 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Nov 01 23:12:17.347969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.347937 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:17.437633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.437564 4981 generic.go:356] "Generic (PLEG): container finished" podID="32cbae16-acbd-43b9-a72c-008458724815" containerID="6871eb6391bd584232989b9ded063e04fae5b807df1e2f180155fe55e88c61e6" exitCode=0 Nov 01 23:12:17.437937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.437848 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" event={"ID":"32cbae16-acbd-43b9-a72c-008458724815","Type":"ContainerDied","Data":"6871eb6391bd584232989b9ded063e04fae5b807df1e2f180155fe55e88c61e6"} Nov 01 23:12:17.440610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.439987 4981 generic.go:356] "Generic (PLEG): container finished" podID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerID="a61d71a69d8bb4581bf90789c8a4771dc1a189deebe192f76389c972902899a8" exitCode=0 Nov 01 23:12:17.440610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.440099 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" event={"ID":"6fa3da20-7cee-42f5-955f-0e90546fadde","Type":"ContainerDied","Data":"a61d71a69d8bb4581bf90789c8a4771dc1a189deebe192f76389c972902899a8"} Nov 01 23:12:17.461046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.459159 4981 generic.go:356] "Generic (PLEG): container finished" podID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerID="2bb0195e003c7f78c178520adb0382ee893bd022ae3d9ff01f0f53bd1517e665" exitCode=0 Nov 01 23:12:17.461046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.459258 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" event={"ID":"cea13bb9-9fdf-448e-b15f-a42b48ea6edb","Type":"ContainerDied","Data":"2bb0195e003c7f78c178520adb0382ee893bd022ae3d9ff01f0f53bd1517e665"} Nov 01 23:12:17.491278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.491208 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:17.491694 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.491667 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" podUID="1428b87e-8632-484b-a440-73a44e8aec59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:17.561228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.561197 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 23:12:17.603320 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.602958 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 23:12:17.606777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.606755 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 23:12:17.617892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.617869 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-dshm\") pod \"32cbae16-acbd-43b9-a72c-008458724815\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " Nov 01 23:12:17.618065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.617899 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-model-cache\") pod \"32cbae16-acbd-43b9-a72c-008458724815\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " Nov 01 23:12:17.618065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.617931 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-model-cache\") pod \"6fa3da20-7cee-42f5-955f-0e90546fadde\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " Nov 01 23:12:17.618065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.617984 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-home\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-home\") pod \"32cbae16-acbd-43b9-a72c-008458724815\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " Nov 01 23:12:17.618065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618014 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-home\") pod \"6fa3da20-7cee-42f5-955f-0e90546fadde\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " Nov 01 23:12:17.618065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618042 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa3da20-7cee-42f5-955f-0e90546fadde-kube-api-access-thmnl\" (UniqueName: \"kubernetes.io/projected/6fa3da20-7cee-42f5-955f-0e90546fadde-kube-api-access-thmnl\") pod \"6fa3da20-7cee-42f5-955f-0e90546fadde\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618087 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tls-certs\" (UniqueName: \"kubernetes.io/secret/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tls-certs\") pod \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618117 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-model-cache\") pod \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618158 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cbae16-acbd-43b9-a72c-008458724815-kube-api-access-2skhq\" (UniqueName: \"kubernetes.io/projected/32cbae16-acbd-43b9-a72c-008458724815-kube-api-access-2skhq\") pod \"32cbae16-acbd-43b9-a72c-008458724815\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618190 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cbae16-acbd-43b9-a72c-008458724815-tls-certs\" (UniqueName: \"kubernetes.io/secret/32cbae16-acbd-43b9-a72c-008458724815-tls-certs\") pod \"32cbae16-acbd-43b9-a72c-008458724815\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618241 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-home\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-home\") pod \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618268 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-dshm\") pod \"6fa3da20-7cee-42f5-955f-0e90546fadde\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618317 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa3da20-7cee-42f5-955f-0e90546fadde-tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3da20-7cee-42f5-955f-0e90546fadde-tls-certs\") pod \"6fa3da20-7cee-42f5-955f-0e90546fadde\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618352 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-kube-api-access-n4jqn\" (UniqueName: \"kubernetes.io/projected/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-kube-api-access-n4jqn\") pod \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " Nov 01 23:12:17.618384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618382 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-tmp-dir\") pod \"6fa3da20-7cee-42f5-955f-0e90546fadde\" (UID: \"6fa3da20-7cee-42f5-955f-0e90546fadde\") " Nov 01 23:12:17.618959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618434 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tmp-dir\") pod \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " Nov 01 23:12:17.618959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618477 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-tmp-dir\") pod \"32cbae16-acbd-43b9-a72c-008458724815\" (UID: \"32cbae16-acbd-43b9-a72c-008458724815\") " Nov 01 23:12:17.618959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.618532 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-dshm\") pod \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\" (UID: \"cea13bb9-9fdf-448e-b15f-a42b48ea6edb\") " Nov 01 23:12:17.619767 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.619740 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-home" pod "6fa3da20-7cee-42f5-955f-0e90546fadde" (UID: "6fa3da20-7cee-42f5-955f-0e90546fadde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.620124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.620103 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-model-cache" pod "32cbae16-acbd-43b9-a72c-008458724815" (UID: "32cbae16-acbd-43b9-a72c-008458724815"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.621263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.620709 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-home" pod "32cbae16-acbd-43b9-a72c-008458724815" (UID: "32cbae16-acbd-43b9-a72c-008458724815"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.621263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.620416 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-model-cache" pod "6fa3da20-7cee-42f5-955f-0e90546fadde" (UID: "6fa3da20-7cee-42f5-955f-0e90546fadde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.621263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.620782 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-tmp-dir" pod "6fa3da20-7cee-42f5-955f-0e90546fadde" (UID: "6fa3da20-7cee-42f5-955f-0e90546fadde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.621263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.620992 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tmp-dir" pod "cea13bb9-9fdf-448e-b15f-a42b48ea6edb" (UID: "cea13bb9-9fdf-448e-b15f-a42b48ea6edb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.621263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.621103 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-home" pod "cea13bb9-9fdf-448e-b15f-a42b48ea6edb" (UID: "cea13bb9-9fdf-448e-b15f-a42b48ea6edb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.621263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.621111 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-tmp-dir" pod "32cbae16-acbd-43b9-a72c-008458724815" (UID: "32cbae16-acbd-43b9-a72c-008458724815"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.623557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.623479 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cbae16-acbd-43b9-a72c-008458724815-kube-api-access-2skhq" pod "32cbae16-acbd-43b9-a72c-008458724815" (UID: "32cbae16-acbd-43b9-a72c-008458724815"). InnerVolumeSpecName "kube-api-access-2skhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:17.629282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.624315 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-model-cache" pod "cea13bb9-9fdf-448e-b15f-a42b48ea6edb" (UID: "cea13bb9-9fdf-448e-b15f-a42b48ea6edb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.629282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.625788 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-dshm" pod "32cbae16-acbd-43b9-a72c-008458724815" (UID: "32cbae16-acbd-43b9-a72c-008458724815"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.629282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.627683 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-dshm" pod "cea13bb9-9fdf-448e-b15f-a42b48ea6edb" (UID: "cea13bb9-9fdf-448e-b15f-a42b48ea6edb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.634997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.634966 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cbae16-acbd-43b9-a72c-008458724815-tls-certs" pod "32cbae16-acbd-43b9-a72c-008458724815" (UID: "32cbae16-acbd-43b9-a72c-008458724815"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:17.643609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.637099 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa3da20-7cee-42f5-955f-0e90546fadde-tls-certs" pod "6fa3da20-7cee-42f5-955f-0e90546fadde" (UID: "6fa3da20-7cee-42f5-955f-0e90546fadde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:17.643609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.637195 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa3da20-7cee-42f5-955f-0e90546fadde-kube-api-access-thmnl" pod "6fa3da20-7cee-42f5-955f-0e90546fadde" (UID: "6fa3da20-7cee-42f5-955f-0e90546fadde"). InnerVolumeSpecName "kube-api-access-thmnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:17.643609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.637307 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-kube-api-access-n4jqn" pod "cea13bb9-9fdf-448e-b15f-a42b48ea6edb" (UID: "cea13bb9-9fdf-448e-b15f-a42b48ea6edb"). InnerVolumeSpecName "kube-api-access-n4jqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:17.643609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.639949 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tls-certs" pod "cea13bb9-9fdf-448e-b15f-a42b48ea6edb" (UID: "cea13bb9-9fdf-448e-b15f-a42b48ea6edb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:17.643609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.642751 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-dshm" pod "6fa3da20-7cee-42f5-955f-0e90546fadde" (UID: "6fa3da20-7cee-42f5-955f-0e90546fadde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:17.720542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720509 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.720542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720538 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720551 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thmnl\" (UniqueName: \"kubernetes.io/projected/6fa3da20-7cee-42f5-955f-0e90546fadde-kube-api-access-thmnl\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720566 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720592 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720605 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skhq\" (UniqueName: \"kubernetes.io/projected/32cbae16-acbd-43b9-a72c-008458724815-kube-api-access-2skhq\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720619 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cbae16-acbd-43b9-a72c-008458724815-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720632 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720643 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720656 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fa3da20-7cee-42f5-955f-0e90546fadde-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720670 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jqn\" (UniqueName: \"kubernetes.io/projected/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-kube-api-access-n4jqn\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720683 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720698 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720711 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720724 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea13bb9-9fdf-448e-b15f-a42b48ea6edb-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720736 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720748 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cbae16-acbd-43b9-a72c-008458724815-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.721096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.720761 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fa3da20-7cee-42f5-955f-0e90546fadde-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:17.974792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:17.974757 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:18.024475 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.024257 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr"] Nov 01 23:12:18.028003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.027976 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7585e6c3-fjntr"] Nov 01 23:12:18.140891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.140858 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:18.276132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.276093 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj"] Nov 01 23:12:18.281684 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.281652 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:18.284308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.284284 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-dc635e-8b2xj"] Nov 01 23:12:18.376766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.376731 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:18.468529 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.468500 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" Nov 01 23:12:18.468529 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.468501 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc" event={"ID":"32cbae16-acbd-43b9-a72c-008458724815","Type":"ContainerDied","Data":"c2f55f56aa3b9850322636804aaeebfd78763ab052c3a911423f496b2d65fd0d"} Nov 01 23:12:18.468802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.468565 4981 scope.go:122] "RemoveContainer" containerID="6871eb6391bd584232989b9ded063e04fae5b807df1e2f180155fe55e88c61e6" Nov 01 23:12:18.471387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.471366 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" Nov 01 23:12:18.471690 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.471659 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z" event={"ID":"6fa3da20-7cee-42f5-955f-0e90546fadde","Type":"ContainerDied","Data":"56540d0d675d5784de5416052e93fe29d09f68445c76b6253affa5fbc4de3958"} Nov 01 23:12:18.473799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.473777 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" event={"ID":"cea13bb9-9fdf-448e-b15f-a42b48ea6edb","Type":"ContainerDied","Data":"27e49209ad7f1df041e5ccdd835f379a0cdc3971be27c06474ebbf6ee84473f7"} Nov 01 23:12:18.474010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.473964 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4" Nov 01 23:12:18.483766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.483740 4981 scope.go:122] "RemoveContainer" containerID="a61d71a69d8bb4581bf90789c8a4771dc1a189deebe192f76389c972902899a8" Nov 01 23:12:18.495849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.495805 4981 scope.go:122] "RemoveContainer" containerID="2bb0195e003c7f78c178520adb0382ee893bd022ae3d9ff01f0f53bd1517e665" Nov 01 23:12:18.541068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.541033 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z"] Nov 01 23:12:18.552952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.552918 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/simulator-w5-e26733-kserve-f668d978b-br58z"] Nov 01 23:12:18.578960 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.578935 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc"] Nov 01 23:12:18.593542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.593516 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/premium-w5-e26733-kserve-5b4c56bd75-694cc"] Nov 01 23:12:18.615832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.615803 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4"] Nov 01 23:12:18.633834 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.633798 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e26733/distinct-2-w5-e26733-kserve-59944cd7d5-g49x4"] Nov 01 23:12:18.733385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.733351 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cbae16-acbd-43b9-a72c-008458724815" path="/var/lib/kubelet/pods/32cbae16-acbd-43b9-a72c-008458724815/volumes" Nov 01 23:12:18.733893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.733812 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" path="/var/lib/kubelet/pods/6fa3da20-7cee-42f5-955f-0e90546fadde/volumes" Nov 01 23:12:18.734205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.734190 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3566773-6869-412b-ac8a-cfe1cd9c3194" path="/var/lib/kubelet/pods/a3566773-6869-412b-ac8a-cfe1cd9c3194/volumes" Nov 01 23:12:18.734903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.734887 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b59b4dad-93de-4dae-a937-7c2c5029d54b" path="/var/lib/kubelet/pods/b59b4dad-93de-4dae-a937-7c2c5029d54b/volumes" Nov 01 23:12:18.735304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.735290 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" path="/var/lib/kubelet/pods/cea13bb9-9fdf-448e-b15f-a42b48ea6edb/volumes" Nov 01 23:12:18.777942 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:18.777866 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:19.366755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:19.366707 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:19.993124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:19.993084 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 23:12:20.052518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.052487 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1428b87e-8632-484b-a440-73a44e8aec59-kube-api-access-gh9zg\" (UniqueName: \"kubernetes.io/projected/1428b87e-8632-484b-a440-73a44e8aec59-kube-api-access-gh9zg\") pod \"1428b87e-8632-484b-a440-73a44e8aec59\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " Nov 01 23:12:20.052688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.052584 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1428b87e-8632-484b-a440-73a44e8aec59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428b87e-8632-484b-a440-73a44e8aec59-plugins-config-volume\") pod \"1428b87e-8632-484b-a440-73a44e8aec59\" (UID: \"1428b87e-8632-484b-a440-73a44e8aec59\") " Nov 01 23:12:20.052912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.052889 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1428b87e-8632-484b-a440-73a44e8aec59-plugins-config-volume" pod "1428b87e-8632-484b-a440-73a44e8aec59" (UID: "1428b87e-8632-484b-a440-73a44e8aec59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:20.053064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.053050 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1428b87e-8632-484b-a440-73a44e8aec59-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.054698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.054674 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1428b87e-8632-484b-a440-73a44e8aec59-kube-api-access-gh9zg" pod "1428b87e-8632-484b-a440-73a44e8aec59" (UID: "1428b87e-8632-484b-a440-73a44e8aec59"). InnerVolumeSpecName "kube-api-access-gh9zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:20.076769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.076748 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 23:12:20.153966 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.153898 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc2e5c9-6387-494d-ae94-f81de624e0f2-kube-api-access-f5gbv\" (UniqueName: \"kubernetes.io/projected/8bc2e5c9-6387-494d-ae94-f81de624e0f2-kube-api-access-f5gbv\") pod \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " Nov 01 23:12:20.154137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.154119 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc2e5c9-6387-494d-ae94-f81de624e0f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc2e5c9-6387-494d-ae94-f81de624e0f2-plugins-config-volume\") pod \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\" (UID: \"8bc2e5c9-6387-494d-ae94-f81de624e0f2\") " Nov 01 23:12:20.154491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.154470 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh9zg\" (UniqueName: \"kubernetes.io/projected/1428b87e-8632-484b-a440-73a44e8aec59-kube-api-access-gh9zg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.154542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.154472 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc2e5c9-6387-494d-ae94-f81de624e0f2-plugins-config-volume" pod "8bc2e5c9-6387-494d-ae94-f81de624e0f2" (UID: "8bc2e5c9-6387-494d-ae94-f81de624e0f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:20.155982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.155965 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc2e5c9-6387-494d-ae94-f81de624e0f2-kube-api-access-f5gbv" pod "8bc2e5c9-6387-494d-ae94-f81de624e0f2" (UID: "8bc2e5c9-6387-494d-ae94-f81de624e0f2"). InnerVolumeSpecName "kube-api-access-f5gbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:20.179785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.179757 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:20.256037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.256004 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bc2e5c9-6387-494d-ae94-f81de624e0f2-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.256037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.256030 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5gbv\" (UniqueName: \"kubernetes.io/projected/8bc2e5c9-6387-494d-ae94-f81de624e0f2-kube-api-access-f5gbv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.496453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.496421 4981 generic.go:356] "Generic (PLEG): container finished" podID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerID="02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4" exitCode=0 Nov 01 23:12:20.496703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.496467 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" Nov 01 23:12:20.496703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.496513 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" event={"ID":"8bc2e5c9-6387-494d-ae94-f81de624e0f2","Type":"ContainerDied","Data":"02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4"} Nov 01 23:12:20.496703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.496560 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp" event={"ID":"8bc2e5c9-6387-494d-ae94-f81de624e0f2","Type":"ContainerDied","Data":"4496c061e506a46368dd23cf2c1a18e4cc006fc6d6000099a60aca66bc13ce59"} Nov 01 23:12:20.496703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.496605 4981 scope.go:122] "RemoveContainer" containerID="02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4" Nov 01 23:12:20.497994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.497974 4981 generic.go:356] "Generic (PLEG): container finished" podID="1428b87e-8632-484b-a440-73a44e8aec59" containerID="ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28" exitCode=0 Nov 01 23:12:20.498115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.498008 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" event={"ID":"1428b87e-8632-484b-a440-73a44e8aec59","Type":"ContainerDied","Data":"ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28"} Nov 01 23:12:20.498115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.498072 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" event={"ID":"1428b87e-8632-484b-a440-73a44e8aec59","Type":"ContainerDied","Data":"9c7a966ac7b529b74b1f82d71a5e095a6b7e654eecf3bd03c3139b68a5edf7b6"} Nov 01 23:12:20.498115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.498022 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd" Nov 01 23:12:20.510284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.510260 4981 scope.go:122] "RemoveContainer" containerID="02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4" Nov 01 23:12:20.510564 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:20.510545 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4\": container with ID starting with 02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4 not found: ID does not exist" containerID="02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4" Nov 01 23:12:20.510660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.510599 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4"} err="failed to get container status \"02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4\": rpc error: code = NotFound desc = could not find container \"02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4\": container with ID starting with 02a2f8f2367e5ab9278921c2cdc83e3906bcf1f49761a755d4cb45ee9522bfb4 not found: ID does not exist" Nov 01 23:12:20.510660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.510620 4981 scope.go:122] "RemoveContainer" containerID="ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28" Nov 01 23:12:20.512961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.512935 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" containerID="cri-o://642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66" gracePeriod=45 Nov 01 23:12:20.521663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.521640 4981 scope.go:122] "RemoveContainer" containerID="ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28" Nov 01 23:12:20.521950 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:20.521929 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28\": container with ID starting with ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28 not found: ID does not exist" containerID="ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28" Nov 01 23:12:20.522006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.521958 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28"} err="failed to get container status \"ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28\": rpc error: code = NotFound desc = could not find container \"ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28\": container with ID starting with ffc292b9e39c701a431508a5d6bca54a2ccaf8b2f2ed318df95d7cf956efce28 not found: ID does not exist" Nov 01 23:12:20.524643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.522682 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp"] Nov 01 23:12:20.526923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.526899 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ee07ad-84779d5f87-xxbvp"] Nov 01 23:12:20.538289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.538266 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" containerID="cri-o://f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02" gracePeriod=45 Nov 01 23:12:20.539943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.539909 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd"] Nov 01 23:12:20.543398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.543376 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ee07ad-7f879ccb6f-24kjd"] Nov 01 23:12:20.726209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.726166 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Nov 01 23:12:20.735490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.735459 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1428b87e-8632-484b-a440-73a44e8aec59" path="/var/lib/kubelet/pods/1428b87e-8632-484b-a440-73a44e8aec59/volumes" Nov 01 23:12:20.736243 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.736215 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" path="/var/lib/kubelet/pods/8bc2e5c9-6387-494d-ae94-f81de624e0f2/volumes" Nov 01 23:12:20.784167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.784142 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 23:12:20.814592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.814544 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 23:12:20.862803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.862774 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-model-cache\") pod \"dae6a304-8020-419b-95ef-5d5242a94e0c\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " Nov 01 23:12:20.862974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.862824 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae6a304-8020-419b-95ef-5d5242a94e0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae6a304-8020-419b-95ef-5d5242a94e0c-tls-certs\") pod \"dae6a304-8020-419b-95ef-5d5242a94e0c\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " Nov 01 23:12:20.862974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.862890 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tmp-dir\") pod \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " Nov 01 23:12:20.862974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.862916 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-home\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-home\") pod \"dae6a304-8020-419b-95ef-5d5242a94e0c\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " Nov 01 23:12:20.862974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.862941 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-model-cache\") pod \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " Nov 01 23:12:20.863223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.862982 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-tmp-dir\") pod \"dae6a304-8020-419b-95ef-5d5242a94e0c\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " Nov 01 23:12:20.863223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863061 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-home\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-home\") pod \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " Nov 01 23:12:20.863223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863090 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tls-certs\") pod \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " Nov 01 23:12:20.863223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863085 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-model-cache" pod "dae6a304-8020-419b-95ef-5d5242a94e0c" (UID: "dae6a304-8020-419b-95ef-5d5242a94e0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.863223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863149 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae6a304-8020-419b-95ef-5d5242a94e0c-kube-api-access-5pb48\" (UniqueName: \"kubernetes.io/projected/dae6a304-8020-419b-95ef-5d5242a94e0c-kube-api-access-5pb48\") pod \"dae6a304-8020-419b-95ef-5d5242a94e0c\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " Nov 01 23:12:20.863223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863210 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-dshm\") pod \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " Nov 01 23:12:20.863540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863240 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-dshm\") pod \"dae6a304-8020-419b-95ef-5d5242a94e0c\" (UID: \"dae6a304-8020-419b-95ef-5d5242a94e0c\") " Nov 01 23:12:20.863540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863273 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-kube-api-access-dtj8l\" (UniqueName: \"kubernetes.io/projected/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-kube-api-access-dtj8l\") pod \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\" (UID: \"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0\") " Nov 01 23:12:20.863540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863287 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-home" pod "dae6a304-8020-419b-95ef-5d5242a94e0c" (UID: "dae6a304-8020-419b-95ef-5d5242a94e0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.863745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863729 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.863816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863749 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.863816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.863775 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tmp-dir" pod "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" (UID: "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.864391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.864364 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-tmp-dir" pod "dae6a304-8020-419b-95ef-5d5242a94e0c" (UID: "dae6a304-8020-419b-95ef-5d5242a94e0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.864391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.864379 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-home" pod "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" (UID: "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.864659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.864634 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-model-cache" pod "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" (UID: "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.866602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.866551 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tls-certs" pod "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" (UID: "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:20.866691 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.866569 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-kube-api-access-dtj8l" pod "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" (UID: "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0"). InnerVolumeSpecName "kube-api-access-dtj8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:20.866843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.866824 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae6a304-8020-419b-95ef-5d5242a94e0c-tls-certs" pod "dae6a304-8020-419b-95ef-5d5242a94e0c" (UID: "dae6a304-8020-419b-95ef-5d5242a94e0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:20.868478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.867545 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-dshm" pod "dae6a304-8020-419b-95ef-5d5242a94e0c" (UID: "dae6a304-8020-419b-95ef-5d5242a94e0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.868478 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.867659 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae6a304-8020-419b-95ef-5d5242a94e0c-kube-api-access-5pb48" pod "dae6a304-8020-419b-95ef-5d5242a94e0c" (UID: "dae6a304-8020-419b-95ef-5d5242a94e0c"). InnerVolumeSpecName "kube-api-access-5pb48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:20.868860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.868837 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-dshm" pod "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" (UID: "24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:20.964348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964316 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae6a304-8020-419b-95ef-5d5242a94e0c-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964346 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964348 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964355 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964364 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964373 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964380 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964390 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pb48\" (UniqueName: \"kubernetes.io/projected/dae6a304-8020-419b-95ef-5d5242a94e0c-kube-api-access-5pb48\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964399 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964408 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae6a304-8020-419b-95ef-5d5242a94e0c-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:20.964593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:20.964416 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtj8l\" (UniqueName: \"kubernetes.io/projected/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0-kube-api-access-dtj8l\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:21.077789 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.077704 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:21.285671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.285611 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" podUID="b153d778-544a-450d-8518-f1536e77e787" containerName="payload-processing" containerID="cri-o://7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0" gracePeriod=25 Nov 01 23:12:21.297145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.297115 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" podUID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" containerName="payload-pre-processing" containerID="cri-o://8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34" gracePeriod=25 Nov 01 23:12:21.339232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.339172 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:21.507699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.507671 4981 generic.go:356] "Generic (PLEG): container finished" podID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerID="f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02" exitCode=0 Nov 01 23:12:21.507901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.507720 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" Nov 01 23:12:21.507901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.507740 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" event={"ID":"dae6a304-8020-419b-95ef-5d5242a94e0c","Type":"ContainerDied","Data":"f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02"} Nov 01 23:12:21.507901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.507774 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx" event={"ID":"dae6a304-8020-419b-95ef-5d5242a94e0c","Type":"ContainerDied","Data":"a80a88afe266bb37afde8377d9988e2bc2d8d06d7c1e0db3af23b45d4e38c97d"} Nov 01 23:12:21.507901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.507793 4981 scope.go:122] "RemoveContainer" containerID="f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02" Nov 01 23:12:21.510340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.510313 4981 generic.go:356] "Generic (PLEG): container finished" podID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerID="642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66" exitCode=0 Nov 01 23:12:21.510468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.510341 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" event={"ID":"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0","Type":"ContainerDied","Data":"642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66"} Nov 01 23:12:21.510468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.510361 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" Nov 01 23:12:21.510468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.510375 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn" event={"ID":"24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0","Type":"ContainerDied","Data":"392c4a1368f59e9496ee80645439540c050a6dd7ac55002cf90d368b7a05d52d"} Nov 01 23:12:21.520816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.520799 4981 scope.go:122] "RemoveContainer" containerID="f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02" Nov 01 23:12:21.521152 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:21.521123 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02\": container with ID starting with f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02 not found: ID does not exist" containerID="f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02" Nov 01 23:12:21.521242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.521162 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02"} err="failed to get container status \"f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02\": rpc error: code = NotFound desc = could not find container \"f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02\": container with ID starting with f745db0c9fbede73fdfbcc2c913c84ec1f4baf3d77adf9a6676fdf98e024ac02 not found: ID does not exist" Nov 01 23:12:21.521242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.521183 4981 scope.go:122] "RemoveContainer" containerID="642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66" Nov 01 23:12:21.530431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.530416 4981 scope.go:122] "RemoveContainer" containerID="642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66" Nov 01 23:12:21.530691 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:21.530675 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66\": container with ID starting with 642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66 not found: ID does not exist" containerID="642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66" Nov 01 23:12:21.530755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.530697 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66"} err="failed to get container status \"642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66\": rpc error: code = NotFound desc = could not find container \"642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66\": container with ID starting with 642c52b012aed4052d406cfca02a23ea0a0954c6d5681ba561c1d6685c3e2d66 not found: ID does not exist" Nov 01 23:12:21.539985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.539965 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx"] Nov 01 23:12:21.543712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.543691 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/premium-w3-6ac182-kserve-576db9bdb-vg5hx"] Nov 01 23:12:21.553778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.553756 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn"] Nov 01 23:12:21.557519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.557500 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6ac182/distinct-2-w3-6ac182-kserve-7f46489b54-7vqwn"] Nov 01 23:12:21.657020 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.656937 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" podUID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:21.657020 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.656941 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" podUID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:21.884513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:21.884483 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:22.574944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:22.574910 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:22.734087 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:22.734053 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" path="/var/lib/kubelet/pods/24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0/volumes" Nov 01 23:12:22.734481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:22.734467 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" path="/var/lib/kubelet/pods/dae6a304-8020-419b-95ef-5d5242a94e0c/volumes" Nov 01 23:12:22.909253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:22.909168 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" podUID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:22.909253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:22.909226 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" podUID="b153d778-544a-450d-8518-f1536e77e787" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:23.360909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:23.360869 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:24.683333 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:24.683297 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" podUID="d815af03-1261-4769-bf33-6a76cf1f17bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:25.382363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:25.382330 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:25.480434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:25.480402 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:26.186175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.186144 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:26.378898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.378866 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:26.468399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.468319 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:26.500294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.500260 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt"] Nov 01 23:12:26.507304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.507274 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e26733-5kwzt"] Nov 01 23:12:26.568713 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.568675 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:26.654213 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.654175 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j"] Nov 01 23:12:26.654468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.654444 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" containerID="cri-o://b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f" gracePeriod=30 Nov 01 23:12:26.696504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.696467 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:26.733184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.733104 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d0b4e8-abed-4565-91b8-f2dde4cea4f9" path="/var/lib/kubelet/pods/48d0b4e8-abed-4565-91b8-f2dde4cea4f9/volumes" Nov 01 23:12:26.988158 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:26.988076 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:27.106308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.106270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:27.250933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.250853 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:27.290787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.290751 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf"] Nov 01 23:12:27.291246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291233 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1428b87e-8632-484b-a440-73a44e8aec59" containerName="payload-processing" Nov 01 23:12:27.291299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291247 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1428b87e-8632-484b-a440-73a44e8aec59" containerName="payload-processing" Nov 01 23:12:27.291299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291268 4981 cpu_manager.go:414] "containerMap: removing container" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" Nov 01 23:12:27.291299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291273 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" Nov 01 23:12:27.291299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291285 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c0911c3c-3b05-4f9b-8de7-5f21dd85c707" containerName="revoke-keys" Nov 01 23:12:27.291299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291291 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0911c3c-3b05-4f9b-8de7-5f21dd85c707" containerName="revoke-keys" Nov 01 23:12:27.291299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291298 4981 cpu_manager.go:414] "containerMap: removing container" podUID="38d1d9b2-08ca-4827-a6ff-b35966fc95ba" containerName="revoke-keys" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291303 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d1d9b2-08ca-4827-a6ff-b35966fc95ba" containerName="revoke-keys" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291314 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291318 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291333 4981 cpu_manager.go:414] "containerMap: removing container" podUID="741ceb40-f226-49d0-b06a-2d1acf59a08f" containerName="revoke-keys" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291339 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741ceb40-f226-49d0-b06a-2d1acf59a08f" containerName="revoke-keys" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291345 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291349 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291357 4981 cpu_manager.go:414] "containerMap: removing container" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291362 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291371 4981 cpu_manager.go:414] "containerMap: removing container" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291375 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291382 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerName="payload-pre-processing" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291386 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerName="payload-pre-processing" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291453 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="741ceb40-f226-49d0-b06a-2d1acf59a08f" containerName="revoke-keys" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291461 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa3da20-7cee-42f5-955f-0e90546fadde" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291467 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="24cdfcf5-5a44-49dd-84b5-5e5db37ec7a0" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291473 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cbae16-acbd-43b9-a72c-008458724815" containerName="main" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291478 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc2e5c9-6387-494d-ae94-f81de624e0f2" containerName="payload-pre-processing" Nov 01 23:12:27.291481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291487 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1428b87e-8632-484b-a440-73a44e8aec59" containerName="payload-processing" Nov 01 23:12:27.292072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291494 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae6a304-8020-419b-95ef-5d5242a94e0c" containerName="main" Nov 01 23:12:27.292072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291501 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0911c3c-3b05-4f9b-8de7-5f21dd85c707" containerName="revoke-keys" Nov 01 23:12:27.292072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291506 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d1d9b2-08ca-4827-a6ff-b35966fc95ba" containerName="revoke-keys" Nov 01 23:12:27.292072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.291512 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea13bb9-9fdf-448e-b15f-a42b48ea6edb" containerName="main" Nov 01 23:12:27.295266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.295244 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.304351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.304326 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf"] Nov 01 23:12:27.335591 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.335542 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:12:27.335757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.335607 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:12:27.338563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.338542 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:12:27.338563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.338553 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:12:27.350217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.350185 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.350345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.350259 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4nwm\" (UniqueName: \"kubernetes.io/projected/8ef156be-e2a1-4291-afa1-366757b0ca2f-kube-api-access-q4nwm\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.350411 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.350390 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.451781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.451744 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.451983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.451838 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.451983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.451894 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4nwm\" (UniqueName: \"kubernetes.io/projected/8ef156be-e2a1-4291-afa1-366757b0ca2f-kube-api-access-q4nwm\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.452914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.452883 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.454559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.454535 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.460223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.460193 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4nwm\" (UniqueName: \"kubernetes.io/projected/8ef156be-e2a1-4291-afa1-366757b0ca2f-kube-api-access-q4nwm\") pod \"maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.560551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.560460 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:27.612219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.612188 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:27.714408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.714378 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:27.770429 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.770384 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst"] Nov 01 23:12:27.773385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.773361 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-6ac182-j8jst"] Nov 01 23:12:27.790896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.790874 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:27.901260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:27.901229 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf"] Nov 01 23:12:27.902528 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:12:27.902501 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef156be_e2a1_4291_afa1_366757b0ca2f.slice/crio-0be98f3c0b81545897544830bf7e4ae01a2e7a99707657bebf81a94618fed74b WatchSource:0}: Error finding container 0be98f3c0b81545897544830bf7e4ae01a2e7a99707657bebf81a94618fed74b: Status 404 returned error can't find the container with id 0be98f3c0b81545897544830bf7e4ae01a2e7a99707657bebf81a94618fed74b Nov 01 23:12:28.022745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.022700 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f"] Nov 01 23:12:28.025634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.025605 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4167bb24-bqk2f"] Nov 01 23:12:28.265096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.265057 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:28.570386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.570289 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" event={"ID":"8ef156be-e2a1-4291-afa1-366757b0ca2f","Type":"ContainerStarted","Data":"50715c1b1cb5e3b3e7f9e6e5c44daa7cab9325b3093f8cedaf15dfd6f1d66671"} Nov 01 23:12:28.570550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.570394 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" event={"ID":"8ef156be-e2a1-4291-afa1-366757b0ca2f","Type":"ContainerStarted","Data":"0be98f3c0b81545897544830bf7e4ae01a2e7a99707657bebf81a94618fed74b"} Nov 01 23:12:28.588339 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.588280 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" podStartSLOduration=1.588267806 podStartE2EDuration="1.588267806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:28.586332862 +0000 UTC m=+2914.014210796" watchObservedRunningTime="2026-11-01 23:12:28.588267806 +0000 UTC m=+2914.016145761" Nov 01 23:12:28.666252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.666218 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:28.734339 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.734305 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21edd4d5-f9e2-4943-9801-861a388ecaee" path="/var/lib/kubelet/pods/21edd4d5-f9e2-4943-9801-861a388ecaee/volumes" Nov 01 23:12:28.734795 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:28.734780 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd4b242-9ffe-44a1-a388-4bf781153174" path="/var/lib/kubelet/pods/2cd4b242-9ffe-44a1-a388-4bf781153174/volumes" Nov 01 23:12:29.375774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:29.375740 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:29.579053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:29.579019 4981 generic.go:356] "Generic (PLEG): container finished" podID="8ef156be-e2a1-4291-afa1-366757b0ca2f" containerID="50715c1b1cb5e3b3e7f9e6e5c44daa7cab9325b3093f8cedaf15dfd6f1d66671" exitCode=0 Nov 01 23:12:29.579208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:29.579107 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" event={"ID":"8ef156be-e2a1-4291-afa1-366757b0ca2f","Type":"ContainerDied","Data":"50715c1b1cb5e3b3e7f9e6e5c44daa7cab9325b3093f8cedaf15dfd6f1d66671"} Nov 01 23:12:30.054587 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.054538 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:30.728049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.728024 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:30.786789 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.786761 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-openshift-ca\") pod \"8ef156be-e2a1-4291-afa1-366757b0ca2f\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " Nov 01 23:12:30.786932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.786825 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef156be-e2a1-4291-afa1-366757b0ca2f-kube-api-access-q4nwm\" (UniqueName: \"kubernetes.io/projected/8ef156be-e2a1-4291-afa1-366757b0ca2f-kube-api-access-q4nwm\") pod \"8ef156be-e2a1-4291-afa1-366757b0ca2f\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " Nov 01 23:12:30.786932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.786887 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-xks-ca\") pod \"8ef156be-e2a1-4291-afa1-366757b0ca2f\" (UID: \"8ef156be-e2a1-4291-afa1-366757b0ca2f\") " Nov 01 23:12:30.787236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.787215 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-openshift-ca" pod "8ef156be-e2a1-4291-afa1-366757b0ca2f" (UID: "8ef156be-e2a1-4291-afa1-366757b0ca2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:30.788952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.788926 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-xks-ca" pod "8ef156be-e2a1-4291-afa1-366757b0ca2f" (UID: "8ef156be-e2a1-4291-afa1-366757b0ca2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:30.789058 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.788934 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef156be-e2a1-4291-afa1-366757b0ca2f-kube-api-access-q4nwm" pod "8ef156be-e2a1-4291-afa1-366757b0ca2f" (UID: "8ef156be-e2a1-4291-afa1-366757b0ca2f"). InnerVolumeSpecName "kube-api-access-q4nwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:30.823423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.823388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:30.888431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.888363 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:30.888431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.888386 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4nwm\" (UniqueName: \"kubernetes.io/projected/8ef156be-e2a1-4291-afa1-366757b0ca2f-kube-api-access-q4nwm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:30.888431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:30.888406 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef156be-e2a1-4291-afa1-366757b0ca2f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:31.450908 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:31.450878 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:31.595311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:31.595281 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" Nov 01 23:12:31.595492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:31.595316 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf" event={"ID":"8ef156be-e2a1-4291-afa1-366757b0ca2f","Type":"ContainerDied","Data":"0be98f3c0b81545897544830bf7e4ae01a2e7a99707657bebf81a94618fed74b"} Nov 01 23:12:31.595492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:31.595364 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0be98f3c0b81545897544830bf7e4ae01a2e7a99707657bebf81a94618fed74b" Nov 01 23:12:31.656187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:31.656152 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" podUID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:31.656360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:31.656152 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" podUID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:31.918187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:31.918162 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 23:12:32.000522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000494 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-socket\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000533 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-kube-api-access-fn7v8\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-kube-api-access-fn7v8\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000601 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-certs\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000646 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-credential-socket\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000700 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-envoy\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000774 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istiod-ca-cert\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000803 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-token\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-token\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000840 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-podinfo\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000848 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-socket" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.000998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000902 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-data\") pod \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\" (UID: \"4a3934c5-9084-4799-b0fe-9b6ac6a054db\") " Nov 01 23:12:32.000998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.000950 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-certs" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.001299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.001206 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-credential-socket" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.001299 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.001232 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istiod-ca-cert" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:32.001400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.001295 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.001400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.001318 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.001400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.001333 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.001520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.001395 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-data" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.002933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.002907 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-envoy" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:32.003046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.002967 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-podinfo" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:32.003294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.003276 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-token" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.003362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.003288 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-kube-api-access-fn7v8" pod "4a3934c5-9084-4799-b0fe-9b6ac6a054db" (UID: "4a3934c5-9084-4799-b0fe-9b6ac6a054db"). InnerVolumeSpecName "kube-api-access-fn7v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.084337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.084246 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx"] Nov 01 23:12:32.093286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.093254 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w"] Nov 01 23:12:32.102786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.102760 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.102786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.102787 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn7v8\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-kube-api-access-fn7v8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.102974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.102802 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.102974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.102815 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.102974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.102826 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.102974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.102839 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3934c5-9084-4799-b0fe-9b6ac6a054db-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:32.187120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.187074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:32.286744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.286696 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:32.611431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.611394 4981 generic.go:356] "Generic (PLEG): container finished" podID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerID="b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f" exitCode=0 Nov 01 23:12:32.611656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.611463 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" Nov 01 23:12:32.611656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.611494 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" event={"ID":"4a3934c5-9084-4799-b0fe-9b6ac6a054db","Type":"ContainerDied","Data":"b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f"} Nov 01 23:12:32.611656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.611609 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j" event={"ID":"4a3934c5-9084-4799-b0fe-9b6ac6a054db","Type":"ContainerDied","Data":"c8dc428cfe02124d1b5579733f47fd5545d795f5a2be3970325b208602e03152"} Nov 01 23:12:32.611656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.611643 4981 scope.go:122] "RemoveContainer" containerID="b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f" Nov 01 23:12:32.640600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.640559 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j"] Nov 01 23:12:32.642746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.642711 4981 scope.go:122] "RemoveContainer" containerID="4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0" Nov 01 23:12:32.644320 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.644299 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-dc635e-openshift-default-f7f6c7ffc-6l42j"] Nov 01 23:12:32.654085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.654054 4981 scope.go:122] "RemoveContainer" containerID="b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f" Nov 01 23:12:32.654448 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:32.654413 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f\": container with ID starting with b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f not found: ID does not exist" containerID="b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f" Nov 01 23:12:32.654604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.654456 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f"} err="failed to get container status \"b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f\": rpc error: code = NotFound desc = could not find container \"b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f\": container with ID starting with b19e716bd1b02c8f675a5f46ec0babaa63180cc8de92324582b6e0351f51a91f not found: ID does not exist" Nov 01 23:12:32.654604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.654476 4981 scope.go:122] "RemoveContainer" containerID="4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0" Nov 01 23:12:32.654820 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:32.654791 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0\": container with ID starting with 4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0 not found: ID does not exist" containerID="4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0" Nov 01 23:12:32.654878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.654831 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0"} err="failed to get container status \"4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0\": rpc error: code = NotFound desc = could not find container \"4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0\": container with ID starting with 4d9ef144b292e21bafbae38933550d7d191363f133768bd695c0c58ad93552c0 not found: ID does not exist" Nov 01 23:12:32.736435 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.736400 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" path="/var/lib/kubelet/pods/4a3934c5-9084-4799-b0fe-9b6ac6a054db/volumes" Nov 01 23:12:32.776836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.776810 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 23:12:32.810066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.810043 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 23:12:32.909766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.909726 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" podUID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:32.909929 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.909758 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" podUID="b153d778-544a-450d-8518-f1536e77e787" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:32.911015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.910997 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-kube-api-access-g2klp\" (UniqueName: \"kubernetes.io/projected/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-kube-api-access-g2klp\") pod \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " Nov 01 23:12:32.911063 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.911044 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa48762-a511-4b63-b3c1-cf90fc387cb4-kube-api-access-6nw67\" (UniqueName: \"kubernetes.io/projected/cfa48762-a511-4b63-b3c1-cf90fc387cb4-kube-api-access-6nw67\") pod \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " Nov 01 23:12:32.911131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.911069 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa48762-a511-4b63-b3c1-cf90fc387cb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa48762-a511-4b63-b3c1-cf90fc387cb4-plugins-config-volume\") pod \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\" (UID: \"cfa48762-a511-4b63-b3c1-cf90fc387cb4\") " Nov 01 23:12:32.911190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.911144 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-plugins-config-volume\") pod \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\" (UID: \"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c\") " Nov 01 23:12:32.911509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.911485 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa48762-a511-4b63-b3c1-cf90fc387cb4-plugins-config-volume" pod "cfa48762-a511-4b63-b3c1-cf90fc387cb4" (UID: "cfa48762-a511-4b63-b3c1-cf90fc387cb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:32.911613 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.911512 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-plugins-config-volume" pod "2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" (UID: "2028d1a7-5b1d-4c6c-b1da-a53c877ad27c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:32.913076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.913050 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-kube-api-access-g2klp" pod "2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" (UID: "2028d1a7-5b1d-4c6c-b1da-a53c877ad27c"). InnerVolumeSpecName "kube-api-access-g2klp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.913194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.913159 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa48762-a511-4b63-b3c1-cf90fc387cb4-kube-api-access-6nw67" pod "cfa48762-a511-4b63-b3c1-cf90fc387cb4" (UID: "cfa48762-a511-4b63-b3c1-cf90fc387cb4"). InnerVolumeSpecName "kube-api-access-6nw67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:32.973842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:32.973806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:33.012397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.012368 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:33.012397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.012393 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2klp\" (UniqueName: \"kubernetes.io/projected/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c-kube-api-access-g2klp\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:33.012397 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.012403 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nw67\" (UniqueName: \"kubernetes.io/projected/cfa48762-a511-4b63-b3c1-cf90fc387cb4-kube-api-access-6nw67\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:33.012637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.012413 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfa48762-a511-4b63-b3c1-cf90fc387cb4-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:33.620990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.620959 4981 generic.go:356] "Generic (PLEG): container finished" podID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" containerID="4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5" exitCode=0 Nov 01 23:12:33.621207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.621018 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" Nov 01 23:12:33.621207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.621049 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" event={"ID":"cfa48762-a511-4b63-b3c1-cf90fc387cb4","Type":"ContainerDied","Data":"4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5"} Nov 01 23:12:33.621207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.621101 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk" event={"ID":"cfa48762-a511-4b63-b3c1-cf90fc387cb4","Type":"ContainerDied","Data":"98dad3c59e4803fab11ef1b32394bff7200ae1f90ee528a3ba328efa55a47015"} Nov 01 23:12:33.621207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.621131 4981 scope.go:122] "RemoveContainer" containerID="4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5" Nov 01 23:12:33.622548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.622529 4981 generic.go:356] "Generic (PLEG): container finished" podID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" containerID="056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657" exitCode=0 Nov 01 23:12:33.622701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.622601 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" Nov 01 23:12:33.622701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.622605 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" event={"ID":"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c","Type":"ContainerDied","Data":"056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657"} Nov 01 23:12:33.622701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.622656 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c" event={"ID":"2028d1a7-5b1d-4c6c-b1da-a53c877ad27c","Type":"ContainerDied","Data":"d8fb48bf21dc994a2f54629c13f52bf8df9b5a855fa616f63937ab7b5b11a0dd"} Nov 01 23:12:33.632096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.631994 4981 scope.go:122] "RemoveContainer" containerID="4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5" Nov 01 23:12:33.632293 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:33.632269 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5\": container with ID starting with 4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5 not found: ID does not exist" containerID="4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5" Nov 01 23:12:33.632405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.632302 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5"} err="failed to get container status \"4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5\": rpc error: code = NotFound desc = could not find container \"4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5\": container with ID starting with 4c0050358414003e8f498a8235705364b349549291d50fc720dca549f57a54f5 not found: ID does not exist" Nov 01 23:12:33.632405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.632326 4981 scope.go:122] "RemoveContainer" containerID="056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657" Nov 01 23:12:33.642756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.642732 4981 scope.go:122] "RemoveContainer" containerID="056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657" Nov 01 23:12:33.643024 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:33.642998 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657\": container with ID starting with 056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657 not found: ID does not exist" containerID="056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657" Nov 01 23:12:33.643127 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.643033 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657"} err="failed to get container status \"056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657\": rpc error: code = NotFound desc = could not find container \"056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657\": container with ID starting with 056550267d1169790cfb7faf61eccc6c16af8b03a7e9e840c62bfa9086288657 not found: ID does not exist" Nov 01 23:12:33.651838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.651815 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c"] Nov 01 23:12:33.657645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.657624 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e26733-679657bdb5-dt48c"] Nov 01 23:12:33.668759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.668738 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk"] Nov 01 23:12:33.673138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.673115 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-e26733-78d8664bcf-2k6bk"] Nov 01 23:12:33.771330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:33.771284 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:34.261837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:34.261799 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:34.357282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:34.357244 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:34.460266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:34.460234 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:34.589944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:34.589869 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:34.682843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:34.682805 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" podUID="d815af03-1261-4769-bf33-6a76cf1f17bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:12:34.733260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:34.733230 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" path="/var/lib/kubelet/pods/2028d1a7-5b1d-4c6c-b1da-a53c877ad27c/volumes" Nov 01 23:12:34.733650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:34.733636 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" path="/var/lib/kubelet/pods/cfa48762-a511-4b63-b3c1-cf90fc387cb4/volumes" Nov 01 23:12:35.260826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:35.260794 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:35.429112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:35.429081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:35.544245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:35.544165 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:35.714791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:35.714763 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm"] Nov 01 23:12:35.715038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:35.715012 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" containerID="cri-o://2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc" gracePeriod=30 Nov 01 23:12:35.753691 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:35.753666 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:35.951227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:35.951200 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 23:12:36.043985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.043953 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d815af03-1261-4769-bf33-6a76cf1f17bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d815af03-1261-4769-bf33-6a76cf1f17bd-plugins-config-volume\") pod \"d815af03-1261-4769-bf33-6a76cf1f17bd\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " Nov 01 23:12:36.044212 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.044044 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d815af03-1261-4769-bf33-6a76cf1f17bd-kube-api-access-9vlsd\" (UniqueName: \"kubernetes.io/projected/d815af03-1261-4769-bf33-6a76cf1f17bd-kube-api-access-9vlsd\") pod \"d815af03-1261-4769-bf33-6a76cf1f17bd\" (UID: \"d815af03-1261-4769-bf33-6a76cf1f17bd\") " Nov 01 23:12:36.044412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.044385 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d815af03-1261-4769-bf33-6a76cf1f17bd-plugins-config-volume" pod "d815af03-1261-4769-bf33-6a76cf1f17bd" (UID: "d815af03-1261-4769-bf33-6a76cf1f17bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:36.046132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.046111 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d815af03-1261-4769-bf33-6a76cf1f17bd-kube-api-access-9vlsd" pod "d815af03-1261-4769-bf33-6a76cf1f17bd" (UID: "d815af03-1261-4769-bf33-6a76cf1f17bd"). InnerVolumeSpecName "kube-api-access-9vlsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:36.058419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.058352 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:36.145312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.145280 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d815af03-1261-4769-bf33-6a76cf1f17bd-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:36.145312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.145308 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vlsd\" (UniqueName: \"kubernetes.io/projected/d815af03-1261-4769-bf33-6a76cf1f17bd-kube-api-access-9vlsd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:36.651036 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.651000 4981 generic.go:356] "Generic (PLEG): container finished" podID="d815af03-1261-4769-bf33-6a76cf1f17bd" containerID="4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef" exitCode=0 Nov 01 23:12:36.651480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.651052 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" Nov 01 23:12:36.651480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.651085 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" event={"ID":"d815af03-1261-4769-bf33-6a76cf1f17bd","Type":"ContainerDied","Data":"4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef"} Nov 01 23:12:36.651480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.651139 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c" event={"ID":"d815af03-1261-4769-bf33-6a76cf1f17bd","Type":"ContainerDied","Data":"991172c4a292d0604b56a2d955d9fb40a36a5a923e12bedef7b3f898eaa22815"} Nov 01 23:12:36.651480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.651160 4981 scope.go:122] "RemoveContainer" containerID="4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef" Nov 01 23:12:36.662909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.662890 4981 scope.go:122] "RemoveContainer" containerID="4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef" Nov 01 23:12:36.663166 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:36.663146 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef\": container with ID starting with 4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef not found: ID does not exist" containerID="4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef" Nov 01 23:12:36.663229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.663178 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef"} err="failed to get container status \"4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef\": rpc error: code = NotFound desc = could not find container \"4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef\": container with ID starting with 4a432859273b299099fa13b9974aa9bfa5bc3db4d8c17fd4a0edb3d5fc69c2ef not found: ID does not exist" Nov 01 23:12:36.683405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.683383 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c"] Nov 01 23:12:36.692856 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.692829 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-6ac182-749cb6cb79-lsg2c"] Nov 01 23:12:36.732382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.732354 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d815af03-1261-4769-bf33-6a76cf1f17bd" path="/var/lib/kubelet/pods/d815af03-1261-4769-bf33-6a76cf1f17bd/volumes" Nov 01 23:12:36.874871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:36.874837 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:37.427385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.427351 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2"] Nov 01 23:12:37.460862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.460831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:37.551192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.551156 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:37.559171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559146 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv"] Nov 01 23:12:37.559798 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559776 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559799 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559837 4981 cpu_manager.go:414] "containerMap: removing container" podUID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" containerName="payload-pre-processing" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559846 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" containerName="payload-pre-processing" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559860 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" containerName="payload-processing" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559869 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" containerName="payload-processing" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559892 4981 cpu_manager.go:414] "containerMap: removing container" podUID="d815af03-1261-4769-bf33-6a76cf1f17bd" containerName="payload-processing" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559899 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d815af03-1261-4769-bf33-6a76cf1f17bd" containerName="payload-processing" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559914 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8ef156be-e2a1-4291-afa1-366757b0ca2f" containerName="revoke-keys" Nov 01 23:12:37.559925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.559920 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef156be-e2a1-4291-afa1-366757b0ca2f" containerName="revoke-keys" Nov 01 23:12:37.560265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.560028 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="d815af03-1261-4769-bf33-6a76cf1f17bd" containerName="payload-processing" Nov 01 23:12:37.560265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.560038 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" Nov 01 23:12:37.560265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.560044 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef156be-e2a1-4291-afa1-366757b0ca2f" containerName="revoke-keys" Nov 01 23:12:37.560265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.560054 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" Nov 01 23:12:37.560265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.560068 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa48762-a511-4b63-b3c1-cf90fc387cb4" containerName="payload-pre-processing" Nov 01 23:12:37.560265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.560074 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2028d1a7-5b1d-4c6c-b1da-a53c877ad27c" containerName="payload-processing" Nov 01 23:12:37.564675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.564657 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.570843 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.570820 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:12:37.571011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.570825 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:12:37.575155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.575133 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv"] Nov 01 23:12:37.610032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.610000 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:12:37.610032 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.610021 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:12:37.663296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.663265 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfgv\" (UniqueName: \"kubernetes.io/projected/ebde4c11-a30e-48af-92bf-943ef2e40fdb-kube-api-access-xvfgv\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.663827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.663303 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.663827 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.663392 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.764624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.764525 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfgv\" (UniqueName: \"kubernetes.io/projected/ebde4c11-a30e-48af-92bf-943ef2e40fdb-kube-api-access-xvfgv\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.764772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.764631 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.764772 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.764730 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.765354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.765334 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.767043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.767014 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.776190 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.776169 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfgv\" (UniqueName: \"kubernetes.io/projected/ebde4c11-a30e-48af-92bf-943ef2e40fdb-kube-api-access-xvfgv\") pod \"maas-api-revoke-sub-7f5539c1-4bkgv\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:37.880524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:37.880492 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:38.155046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:38.155018 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv"] Nov 01 23:12:38.155925 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:12:38.155901 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebde4c11_a30e_48af_92bf_943ef2e40fdb.slice/crio-fdc6b1bc1a1e98370886bca63e5824ebf7c1d975a85b6535249700f3fa380bc2 WatchSource:0}: Error finding container fdc6b1bc1a1e98370886bca63e5824ebf7c1d975a85b6535249700f3fa380bc2: Status 404 returned error can't find the container with id fdc6b1bc1a1e98370886bca63e5824ebf7c1d975a85b6535249700f3fa380bc2 Nov 01 23:12:38.670432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:38.670391 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" event={"ID":"ebde4c11-a30e-48af-92bf-943ef2e40fdb","Type":"ContainerStarted","Data":"4692cd6c3a4c2ae6876f019287e916f6e961199fd9fce38d5b1f12b0f0e91237"} Nov 01 23:12:38.670930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:38.670448 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" event={"ID":"ebde4c11-a30e-48af-92bf-943ef2e40fdb","Type":"ContainerStarted","Data":"fdc6b1bc1a1e98370886bca63e5824ebf7c1d975a85b6535249700f3fa380bc2"} Nov 01 23:12:38.688924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:38.688879 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" podStartSLOduration=1.6888643700000001 podStartE2EDuration="1.68886437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:38.685744619 +0000 UTC m=+2924.113622550" watchObservedRunningTime="2026-11-01 23:12:38.68886437 +0000 UTC m=+2924.116742350" Nov 01 23:12:38.880880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:38.880802 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:38.981303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:38.981270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:39.023907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:39.023874 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn"] Nov 01 23:12:39.026380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:39.026359 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c75f8a-btnvn"] Nov 01 23:12:39.356288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:39.356257 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:39.455933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:39.455902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:39.679676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:39.679644 4981 generic.go:356] "Generic (PLEG): container finished" podID="ebde4c11-a30e-48af-92bf-943ef2e40fdb" containerID="4692cd6c3a4c2ae6876f019287e916f6e961199fd9fce38d5b1f12b0f0e91237" exitCode=0 Nov 01 23:12:39.680097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:39.679730 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" event={"ID":"ebde4c11-a30e-48af-92bf-943ef2e40fdb","Type":"ContainerDied","Data":"4692cd6c3a4c2ae6876f019287e916f6e961199fd9fce38d5b1f12b0f0e91237"} Nov 01 23:12:40.067795 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.067722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:40.631147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.631111 4981 patch_prober.go:29] interesting pod/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:12:40.631324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.631165 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:12:40.733736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.733704 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9027f1c-3f66-4084-b199-bacccc6e0a67" path="/var/lib/kubelet/pods/c9027f1c-3f66-4084-b199-bacccc6e0a67/volumes" Nov 01 23:12:40.842919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.842892 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:40.980652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.980630 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 23:12:40.999936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.999910 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-openshift-ca\") pod \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " Nov 01 23:12:41.000054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:40.999957 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-xks-ca\") pod \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " Nov 01 23:12:41.000098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.000073 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebde4c11-a30e-48af-92bf-943ef2e40fdb-kube-api-access-xvfgv\" (UniqueName: \"kubernetes.io/projected/ebde4c11-a30e-48af-92bf-943ef2e40fdb-kube-api-access-xvfgv\") pod \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\" (UID: \"ebde4c11-a30e-48af-92bf-943ef2e40fdb\") " Nov 01 23:12:41.000275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.000250 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-openshift-ca" pod "ebde4c11-a30e-48af-92bf-943ef2e40fdb" (UID: "ebde4c11-a30e-48af-92bf-943ef2e40fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:41.001281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.001260 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.002930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.002901 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-xks-ca" pod "ebde4c11-a30e-48af-92bf-943ef2e40fdb" (UID: "ebde4c11-a30e-48af-92bf-943ef2e40fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:41.003969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.003688 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebde4c11-a30e-48af-92bf-943ef2e40fdb-kube-api-access-xvfgv" pod "ebde4c11-a30e-48af-92bf-943ef2e40fdb" (UID: "ebde4c11-a30e-48af-92bf-943ef2e40fdb"). InnerVolumeSpecName "kube-api-access-xvfgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.101831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.101796 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-podinfo\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.101847 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-credential-socket\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.101894 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istiod-ca-cert\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.101926 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-kube-api-access-zgfjs\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-kube-api-access-zgfjs\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.101952 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-token\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-token\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102019 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.101980 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-certs\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102021 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-socket\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102229 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-credential-socket" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.102344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102266 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-envoy\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102334 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-certs" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.102628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102359 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-socket" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.102628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102357 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istiod-ca-cert" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:41.102628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102403 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-data\") pod \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\" (UID: \"9775d38d-d819-4c0d-a5a4-5c006a9c2511\") " Nov 01 23:12:41.102819 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102726 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-data" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.102879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102828 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.102879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102847 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.102879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102863 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfgv\" (UniqueName: \"kubernetes.io/projected/ebde4c11-a30e-48af-92bf-943ef2e40fdb-kube-api-access-xvfgv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.103072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102880 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.103072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102893 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebde4c11-a30e-48af-92bf-943ef2e40fdb-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.103072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102902 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.103072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.102911 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.104234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.104210 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-kube-api-access-zgfjs" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "kube-api-access-zgfjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.104365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.104254 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-token" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.104532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.104516 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-envoy" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:41.104629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.104529 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-podinfo" pod "9775d38d-d819-4c0d-a5a4-5c006a9c2511" (UID: "9775d38d-d819-4c0d-a5a4-5c006a9c2511"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:12:41.203545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.203518 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgfjs\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-kube-api-access-zgfjs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.203545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.203545 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.203733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.203555 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.203733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.203563 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9775d38d-d819-4c0d-a5a4-5c006a9c2511-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.566280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.566253 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 23:12:41.569253 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.569233 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 23:12:41.697174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.697142 4981 generic.go:356] "Generic (PLEG): container finished" podID="b153d778-544a-450d-8518-f1536e77e787" containerID="7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0" exitCode=0 Nov 01 23:12:41.697360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.697233 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" event={"ID":"b153d778-544a-450d-8518-f1536e77e787","Type":"ContainerDied","Data":"7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0"} Nov 01 23:12:41.697360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.697331 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" event={"ID":"b153d778-544a-450d-8518-f1536e77e787","Type":"ContainerDied","Data":"56d7752d337c5884ddea2eb72680ac281ed746b4995161af8a9f94bca6bec208"} Nov 01 23:12:41.697522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.697362 4981 scope.go:122] "RemoveContainer" containerID="7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0" Nov 01 23:12:41.697670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.697220 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5" Nov 01 23:12:41.699279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.699261 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" Nov 01 23:12:41.699392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.699291 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv" event={"ID":"ebde4c11-a30e-48af-92bf-943ef2e40fdb","Type":"ContainerDied","Data":"fdc6b1bc1a1e98370886bca63e5824ebf7c1d975a85b6535249700f3fa380bc2"} Nov 01 23:12:41.699392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.699339 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc6b1bc1a1e98370886bca63e5824ebf7c1d975a85b6535249700f3fa380bc2" Nov 01 23:12:41.701076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.701056 4981 generic.go:356] "Generic (PLEG): container finished" podID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" containerID="8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34" exitCode=0 Nov 01 23:12:41.701196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.701113 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" Nov 01 23:12:41.701196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.701142 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" event={"ID":"41289d4e-dbd9-425d-8f5e-e46cd1baa791","Type":"ContainerDied","Data":"8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34"} Nov 01 23:12:41.701305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.701198 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd" event={"ID":"41289d4e-dbd9-425d-8f5e-e46cd1baa791","Type":"ContainerDied","Data":"8ad8753cf5979179ac6bf5003b4e656e76d6d0c338eb536d32d7510e7a098fa9"} Nov 01 23:12:41.702847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.702824 4981 generic.go:356] "Generic (PLEG): container finished" podID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerID="2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc" exitCode=0 Nov 01 23:12:41.706619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.702970 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" event={"ID":"9775d38d-d819-4c0d-a5a4-5c006a9c2511","Type":"ContainerDied","Data":"2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc"} Nov 01 23:12:41.706619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.702992 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" Nov 01 23:12:41.706619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.703018 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm" event={"ID":"9775d38d-d819-4c0d-a5a4-5c006a9c2511","Type":"ContainerDied","Data":"c977872441fa8ee833483e0870122dcbb5df1a4936b3ddc690163ccbe6da2744"} Nov 01 23:12:41.709157 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.709142 4981 scope.go:122] "RemoveContainer" containerID="7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0" Nov 01 23:12:41.709454 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:41.709429 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0\": container with ID starting with 7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0 not found: ID does not exist" containerID="7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0" Nov 01 23:12:41.709540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.709466 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0"} err="failed to get container status \"7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0\": rpc error: code = NotFound desc = could not find container \"7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0\": container with ID starting with 7cd9b65b8bf3d6d2c30f8137659a62c59b623fe6463cae1ba7247569d2dc60a0 not found: ID does not exist" Nov 01 23:12:41.709540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.709490 4981 scope.go:122] "RemoveContainer" containerID="8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34" Nov 01 23:12:41.710251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.710091 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b153d778-544a-450d-8518-f1536e77e787-kube-api-access-hvwn8\" (UniqueName: \"kubernetes.io/projected/b153d778-544a-450d-8518-f1536e77e787-kube-api-access-hvwn8\") pod \"b153d778-544a-450d-8518-f1536e77e787\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " Nov 01 23:12:41.710251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.710148 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41289d4e-dbd9-425d-8f5e-e46cd1baa791-kube-api-access-m4rv4\" (UniqueName: \"kubernetes.io/projected/41289d4e-dbd9-425d-8f5e-e46cd1baa791-kube-api-access-m4rv4\") pod \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " Nov 01 23:12:41.710251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.710246 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41289d4e-dbd9-425d-8f5e-e46cd1baa791-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41289d4e-dbd9-425d-8f5e-e46cd1baa791-plugins-config-volume\") pod \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\" (UID: \"41289d4e-dbd9-425d-8f5e-e46cd1baa791\") " Nov 01 23:12:41.710469 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.710299 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b153d778-544a-450d-8518-f1536e77e787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b153d778-544a-450d-8518-f1536e77e787-plugins-config-volume\") pod \"b153d778-544a-450d-8518-f1536e77e787\" (UID: \"b153d778-544a-450d-8518-f1536e77e787\") " Nov 01 23:12:41.710839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.710815 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41289d4e-dbd9-425d-8f5e-e46cd1baa791-plugins-config-volume" pod "41289d4e-dbd9-425d-8f5e-e46cd1baa791" (UID: "41289d4e-dbd9-425d-8f5e-e46cd1baa791"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:41.710933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.710814 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b153d778-544a-450d-8518-f1536e77e787-plugins-config-volume" pod "b153d778-544a-450d-8518-f1536e77e787" (UID: "b153d778-544a-450d-8518-f1536e77e787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:41.712697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.712614 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b153d778-544a-450d-8518-f1536e77e787-kube-api-access-hvwn8" pod "b153d778-544a-450d-8518-f1536e77e787" (UID: "b153d778-544a-450d-8518-f1536e77e787"). InnerVolumeSpecName "kube-api-access-hvwn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.712801 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.712703 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41289d4e-dbd9-425d-8f5e-e46cd1baa791-kube-api-access-m4rv4" pod "41289d4e-dbd9-425d-8f5e-e46cd1baa791" (UID: "41289d4e-dbd9-425d-8f5e-e46cd1baa791"). InnerVolumeSpecName "kube-api-access-m4rv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:41.726976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.726953 4981 scope.go:122] "RemoveContainer" containerID="8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34" Nov 01 23:12:41.727288 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:41.727267 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34\": container with ID starting with 8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34 not found: ID does not exist" containerID="8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34" Nov 01 23:12:41.727369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.727299 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34"} err="failed to get container status \"8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34\": rpc error: code = NotFound desc = could not find container \"8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34\": container with ID starting with 8b6a489d05c2477a5474f401796839b8d0de3fdc638661f3dca2da2f537b4d34 not found: ID does not exist" Nov 01 23:12:41.727369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.727326 4981 scope.go:122] "RemoveContainer" containerID="2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc" Nov 01 23:12:41.731561 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.731531 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm"] Nov 01 23:12:41.737919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.737894 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6ac182-openshift-default-59f54bdb8f-nvgbm"] Nov 01 23:12:41.742479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.742461 4981 scope.go:122] "RemoveContainer" containerID="2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc" Nov 01 23:12:41.742818 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:41.742797 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc\": container with ID starting with 2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc not found: ID does not exist" containerID="2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc" Nov 01 23:12:41.742907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.742826 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc"} err="failed to get container status \"2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc\": rpc error: code = NotFound desc = could not find container \"2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc\": container with ID starting with 2c0fc1542e0925b8c321a1617126e1651ae33f42e1039b8000b19e3522e295cc not found: ID does not exist" Nov 01 23:12:41.812073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.812035 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41289d4e-dbd9-425d-8f5e-e46cd1baa791-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.812073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.812067 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b153d778-544a-450d-8518-f1536e77e787-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.812073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.812078 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvwn8\" (UniqueName: \"kubernetes.io/projected/b153d778-544a-450d-8518-f1536e77e787-kube-api-access-hvwn8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:41.812315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:41.812086 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rv4\" (UniqueName: \"kubernetes.io/projected/41289d4e-dbd9-425d-8f5e-e46cd1baa791-kube-api-access-m4rv4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:42.024611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.024584 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5"] Nov 01 23:12:42.028426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.028403 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-dc635e-56c4f45f6c-2p9r5"] Nov 01 23:12:42.043433 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.043409 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd"] Nov 01 23:12:42.048286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.048266 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-dc635e-545c947cd7-7hwpd"] Nov 01 23:12:42.427777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.427743 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" podUID="f58e78c4-ffa9-4957-985d-682836f68224" containerName="payload-pre-processing" containerID="cri-o://a6c7f389cd73760f8cd5526ee216ca8967ba9b2b09100c9c0eacf30e1505e367" gracePeriod=25 Nov 01 23:12:42.632701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.632672 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:42.729323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.729284 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:42.734961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.734935 4981 generic.go:356] "Generic (PLEG): container finished" podID="f58e78c4-ffa9-4957-985d-682836f68224" containerID="a6c7f389cd73760f8cd5526ee216ca8967ba9b2b09100c9c0eacf30e1505e367" exitCode=0 Nov 01 23:12:42.735117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.734946 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" path="/var/lib/kubelet/pods/41289d4e-dbd9-425d-8f5e-e46cd1baa791/volumes" Nov 01 23:12:42.735591 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.735553 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" path="/var/lib/kubelet/pods/9775d38d-d819-4c0d-a5a4-5c006a9c2511/volumes" Nov 01 23:12:42.736055 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.736038 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b153d778-544a-450d-8518-f1536e77e787" path="/var/lib/kubelet/pods/b153d778-544a-450d-8518-f1536e77e787/volumes" Nov 01 23:12:42.736358 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.736338 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" event={"ID":"f58e78c4-ffa9-4957-985d-682836f68224","Type":"ContainerDied","Data":"a6c7f389cd73760f8cd5526ee216ca8967ba9b2b09100c9c0eacf30e1505e367"} Nov 01 23:12:42.749627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.749605 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 23:12:42.927918 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.927887 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58e78c4-ffa9-4957-985d-682836f68224-kube-api-access-jdfcm\" (UniqueName: \"kubernetes.io/projected/f58e78c4-ffa9-4957-985d-682836f68224-kube-api-access-jdfcm\") pod \"f58e78c4-ffa9-4957-985d-682836f68224\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " Nov 01 23:12:42.928078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.927939 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58e78c4-ffa9-4957-985d-682836f68224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e78c4-ffa9-4957-985d-682836f68224-plugins-config-volume\") pod \"f58e78c4-ffa9-4957-985d-682836f68224\" (UID: \"f58e78c4-ffa9-4957-985d-682836f68224\") " Nov 01 23:12:42.928354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.928331 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58e78c4-ffa9-4957-985d-682836f68224-plugins-config-volume" pod "f58e78c4-ffa9-4957-985d-682836f68224" (UID: "f58e78c4-ffa9-4957-985d-682836f68224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:12:42.930162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:42.930138 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58e78c4-ffa9-4957-985d-682836f68224-kube-api-access-jdfcm" pod "f58e78c4-ffa9-4957-985d-682836f68224" (UID: "f58e78c4-ffa9-4957-985d-682836f68224"). InnerVolumeSpecName "kube-api-access-jdfcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:43.028987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.028889 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdfcm\" (UniqueName: \"kubernetes.io/projected/f58e78c4-ffa9-4957-985d-682836f68224-kube-api-access-jdfcm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:43.028987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.028928 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e78c4-ffa9-4957-985d-682836f68224-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:43.747854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.747824 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" Nov 01 23:12:43.748073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.747826 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2" event={"ID":"f58e78c4-ffa9-4957-985d-682836f68224","Type":"ContainerDied","Data":"fa069e642f1889a847624e55976bcffd87adf92d775abc66e4d17830dde48254"} Nov 01 23:12:43.748073 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.747968 4981 scope.go:122] "RemoveContainer" containerID="a6c7f389cd73760f8cd5526ee216ca8967ba9b2b09100c9c0eacf30e1505e367" Nov 01 23:12:43.770045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.770018 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2"] Nov 01 23:12:43.773953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.773930 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e5f47e-677cd99f5f-vbxq2"] Nov 01 23:12:43.846919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:43.846878 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:44.021431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:44.021106 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq"] Nov 01 23:12:44.024161 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:44.024134 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs"] Nov 01 23:12:44.026664 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:44.026602 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3c43142-29bjq"] Nov 01 23:12:44.029048 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:44.029027 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99942e3a-5v4hs"] Nov 01 23:12:44.523290 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:44.523258 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:12:44.732953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:44.732918 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66e0b29-bc68-41f2-b661-1a5a44ce4596" path="/var/lib/kubelet/pods/b66e0b29-bc68-41f2-b661-1a5a44ce4596/volumes" Nov 01 23:12:44.733343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:44.733330 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7347f47-2381-421c-81b5-2b1109615b48" path="/var/lib/kubelet/pods/d7347f47-2381-421c-81b5-2b1109615b48/volumes" Nov 01 23:12:44.733693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:44.733681 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58e78c4-ffa9-4957-985d-682836f68224" path="/var/lib/kubelet/pods/f58e78c4-ffa9-4957-985d-682836f68224/volumes" Nov 01 23:12:45.695981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:45.695950 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:47.106037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.105977 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" containerID="cri-o://2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc" gracePeriod=45 Nov 01 23:12:47.113987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.113946 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" containerID="cri-o://33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269" gracePeriod=45 Nov 01 23:12:47.201924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.201889 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": dial tcp 10.134.0.58:8000: connect: connection refused" Nov 01 23:12:47.253403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.253352 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:47.375543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.375476 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 23:12:47.393644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.393621 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 23:12:47.572289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572251 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-home\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-home\") pod \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " Nov 01 23:12:47.572470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572299 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-dshm\") pod \"ff04bd24-bb61-462b-acf7-1b11637a0eff\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " Nov 01 23:12:47.572470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572330 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff04bd24-bb61-462b-acf7-1b11637a0eff-kube-api-access-d94rv\" (UniqueName: \"kubernetes.io/projected/ff04bd24-bb61-462b-acf7-1b11637a0eff-kube-api-access-d94rv\") pod \"ff04bd24-bb61-462b-acf7-1b11637a0eff\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " Nov 01 23:12:47.572470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572367 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-model-cache\") pod \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " Nov 01 23:12:47.572470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572414 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-model-cache\") pod \"ff04bd24-bb61-462b-acf7-1b11637a0eff\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572476 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1794effc-dda4-4f3c-a529-5fa2fc655d6b-kube-api-access-kz76w\" (UniqueName: \"kubernetes.io/projected/1794effc-dda4-4f3c-a529-5fa2fc655d6b-kube-api-access-kz76w\") pod \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572523 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tmp-dir\") pod \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572571 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-home" pod "1794effc-dda4-4f3c-a529-5fa2fc655d6b" (UID: "1794effc-dda4-4f3c-a529-5fa2fc655d6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572615 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tls-certs\") pod \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572626 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-model-cache" pod "1794effc-dda4-4f3c-a529-5fa2fc655d6b" (UID: "1794effc-dda4-4f3c-a529-5fa2fc655d6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572647 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-dshm\") pod \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\" (UID: \"1794effc-dda4-4f3c-a529-5fa2fc655d6b\") " Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572675 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff04bd24-bb61-462b-acf7-1b11637a0eff-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff04bd24-bb61-462b-acf7-1b11637a0eff-tls-certs\") pod \"ff04bd24-bb61-462b-acf7-1b11637a0eff\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572736 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-model-cache" pod "ff04bd24-bb61-462b-acf7-1b11637a0eff" (UID: "ff04bd24-bb61-462b-acf7-1b11637a0eff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.572763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572758 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-tmp-dir\") pod \"ff04bd24-bb61-462b-acf7-1b11637a0eff\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " Nov 01 23:12:47.573264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572794 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-home\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-home\") pod \"ff04bd24-bb61-462b-acf7-1b11637a0eff\" (UID: \"ff04bd24-bb61-462b-acf7-1b11637a0eff\") " Nov 01 23:12:47.573264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.572841 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tmp-dir" pod "1794effc-dda4-4f3c-a529-5fa2fc655d6b" (UID: "1794effc-dda4-4f3c-a529-5fa2fc655d6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.573264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.573119 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.573264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.573137 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.573264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.573151 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.573264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.573164 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.573557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.573396 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-home" pod "ff04bd24-bb61-462b-acf7-1b11637a0eff" (UID: "ff04bd24-bb61-462b-acf7-1b11637a0eff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.573557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.573459 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-tmp-dir" pod "ff04bd24-bb61-462b-acf7-1b11637a0eff" (UID: "ff04bd24-bb61-462b-acf7-1b11637a0eff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.575281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.575253 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-dshm" pod "ff04bd24-bb61-462b-acf7-1b11637a0eff" (UID: "ff04bd24-bb61-462b-acf7-1b11637a0eff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.575456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.575431 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-dshm" pod "1794effc-dda4-4f3c-a529-5fa2fc655d6b" (UID: "1794effc-dda4-4f3c-a529-5fa2fc655d6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:12:47.575532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.575438 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1794effc-dda4-4f3c-a529-5fa2fc655d6b-kube-api-access-kz76w" pod "1794effc-dda4-4f3c-a529-5fa2fc655d6b" (UID: "1794effc-dda4-4f3c-a529-5fa2fc655d6b"). InnerVolumeSpecName "kube-api-access-kz76w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:47.575532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.575458 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tls-certs" pod "1794effc-dda4-4f3c-a529-5fa2fc655d6b" (UID: "1794effc-dda4-4f3c-a529-5fa2fc655d6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:47.575532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.575471 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff04bd24-bb61-462b-acf7-1b11637a0eff-tls-certs" pod "ff04bd24-bb61-462b-acf7-1b11637a0eff" (UID: "ff04bd24-bb61-462b-acf7-1b11637a0eff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:12:47.575532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.575459 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff04bd24-bb61-462b-acf7-1b11637a0eff-kube-api-access-d94rv" pod "ff04bd24-bb61-462b-acf7-1b11637a0eff" (UID: "ff04bd24-bb61-462b-acf7-1b11637a0eff"). InnerVolumeSpecName "kube-api-access-d94rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:12:47.673925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673887 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.673925 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673928 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94rv\" (UniqueName: \"kubernetes.io/projected/ff04bd24-bb61-462b-acf7-1b11637a0eff-kube-api-access-d94rv\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.674131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673939 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz76w\" (UniqueName: \"kubernetes.io/projected/1794effc-dda4-4f3c-a529-5fa2fc655d6b-kube-api-access-kz76w\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.674131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673948 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1794effc-dda4-4f3c-a529-5fa2fc655d6b-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.674131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673957 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1794effc-dda4-4f3c-a529-5fa2fc655d6b-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.674131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673964 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff04bd24-bb61-462b-acf7-1b11637a0eff-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.674131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673972 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.674131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.673980 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff04bd24-bb61-462b-acf7-1b11637a0eff-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:12:47.779237 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.779202 4981 generic.go:356] "Generic (PLEG): container finished" podID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerID="2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc" exitCode=0 Nov 01 23:12:47.779421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.779249 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" Nov 01 23:12:47.779421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.779283 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" event={"ID":"ff04bd24-bb61-462b-acf7-1b11637a0eff","Type":"ContainerDied","Data":"2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc"} Nov 01 23:12:47.779421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.779318 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" event={"ID":"ff04bd24-bb61-462b-acf7-1b11637a0eff","Type":"ContainerDied","Data":"30a58db206c30125822def58472d757ad8b704730098979cfb7a2a29b086d6b9"} Nov 01 23:12:47.779421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.779336 4981 scope.go:122] "RemoveContainer" containerID="2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc" Nov 01 23:12:47.780704 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.780683 4981 generic.go:356] "Generic (PLEG): container finished" podID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerID="33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269" exitCode=0 Nov 01 23:12:47.780838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.780717 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" event={"ID":"1794effc-dda4-4f3c-a529-5fa2fc655d6b","Type":"ContainerDied","Data":"33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269"} Nov 01 23:12:47.780838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.780753 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" event={"ID":"1794effc-dda4-4f3c-a529-5fa2fc655d6b","Type":"ContainerDied","Data":"167aaee2ed54c27ac576c2441c10a2bb1747052c462de931e750289d7e20e6b1"} Nov 01 23:12:47.780838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.780756 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w" Nov 01 23:12:47.791533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.791517 4981 scope.go:122] "RemoveContainer" containerID="2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc" Nov 01 23:12:47.791813 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:47.791796 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc\": container with ID starting with 2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc not found: ID does not exist" containerID="2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc" Nov 01 23:12:47.791873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.791820 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc"} err="failed to get container status \"2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc\": rpc error: code = NotFound desc = could not find container \"2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc\": container with ID starting with 2293ba97f6aec3537e28671275c431386ca79663d32eb258bb4873b89ddbefbc not found: ID does not exist" Nov 01 23:12:47.791873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.791837 4981 scope.go:122] "RemoveContainer" containerID="33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269" Nov 01 23:12:47.801832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.801806 4981 scope.go:122] "RemoveContainer" containerID="33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269" Nov 01 23:12:47.802103 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:12:47.802082 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269\": container with ID starting with 33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269 not found: ID does not exist" containerID="33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269" Nov 01 23:12:47.802173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.802114 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269"} err="failed to get container status \"33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269\": rpc error: code = NotFound desc = could not find container \"33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269\": container with ID starting with 33e3e5e3f10c88269aff7a54ec739fb7baf8ad8aa516d16437b20342343be269 not found: ID does not exist" Nov 01 23:12:47.807313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.807290 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx"] Nov 01 23:12:47.813875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.813854 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx"] Nov 01 23:12:47.824367 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.824347 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w"] Nov 01 23:12:47.828794 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:47.828772 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e5f47e/premium-w0-e5f47e-kserve-856fcf6b7d-mhh7w"] Nov 01 23:12:48.314682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:48.314643 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e5f47e/distinct-w0-e5f47e-kserve-6fc77fff85-rbdxx" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:12:48.733137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:48.733107 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" path="/var/lib/kubelet/pods/1794effc-dda4-4f3c-a529-5fa2fc655d6b/volumes" Nov 01 23:12:48.733563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:48.733548 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" path="/var/lib/kubelet/pods/ff04bd24-bb61-462b-acf7-1b11637a0eff/volumes" Nov 01 23:12:49.527493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:49.527465 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:52.671140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:52.671098 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:52.819981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:52.819951 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:52.826631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:52.826603 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf"] Nov 01 23:12:52.831001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:52.830970 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-e5f47e-qnfzf"] Nov 01 23:12:52.920961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:52.920929 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:53.318482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:53.318451 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:54.732816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:54.732782 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef156be-e2a1-4291-afa1-366757b0ca2f" path="/var/lib/kubelet/pods/8ef156be-e2a1-4291-afa1-366757b0ca2f/volumes" Nov 01 23:12:59.223561 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:59.223531 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:59.325179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:59.325150 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:12:59.548393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:59.548307 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj"] Nov 01 23:12:59.548608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:59.548559 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" containerID="cri-o://f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe" gracePeriod=30 Nov 01 23:12:59.571753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:12:59.571725 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:00.687883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:00.687851 4981 patch_prober.go:29] interesting pod/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:13:00.688254 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:00.687917 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:13:03.333176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:03.333149 4981 scope.go:122] "RemoveContainer" containerID="fe48403743142f1823ea250d53c85ff4eed58e3fa0372302281e694f6bbafb1c" Nov 01 23:13:03.344505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:03.344483 4981 scope.go:122] "RemoveContainer" containerID="5697bf827521e8bc098ff381498d633d5c9f26496697645707bddf38cb5198d6" Nov 01 23:13:03.355716 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:03.355699 4981 scope.go:122] "RemoveContainer" containerID="ae4aacdda4ea3a9b5e0b5ca655e8dfcbf93ffd966a484b4d5cffec0853896a04" Nov 01 23:13:03.367123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:03.367102 4981 scope.go:122] "RemoveContainer" containerID="0b5bac2aa770efc732ef1d5075afa97c900c518c39a8f8e6c3acecb2078796f4" Nov 01 23:13:03.376384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:03.376367 4981 scope.go:122] "RemoveContainer" containerID="befe779c5945cd508efa960509fa8bdf163f3ad1eb6a7f5c664a5a004758e040" Nov 01 23:13:04.830540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.830508 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 23:13:04.906910 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.906879 4981 generic.go:356] "Generic (PLEG): container finished" podID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerID="f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe" exitCode=0 Nov 01 23:13:04.907116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.906931 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" Nov 01 23:13:04.907116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.906976 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" event={"ID":"e4704702-163d-464d-b7fa-c0723a4ba01b","Type":"ContainerDied","Data":"f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe"} Nov 01 23:13:04.907116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.907040 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj" event={"ID":"e4704702-163d-464d-b7fa-c0723a4ba01b","Type":"ContainerDied","Data":"fc32040fbc8c8a5e7e77aef52c73b018d495328d5531c5fba492a8ba4a2c6732"} Nov 01 23:13:04.907116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.907074 4981 scope.go:122] "RemoveContainer" containerID="f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe" Nov 01 23:13:04.917488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.917412 4981 scope.go:122] "RemoveContainer" containerID="f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe" Nov 01 23:13:04.917801 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:13:04.917774 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe\": container with ID starting with f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe not found: ID does not exist" containerID="f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe" Nov 01 23:13:04.917885 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.917814 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe"} err="failed to get container status \"f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe\": rpc error: code = NotFound desc = could not find container \"f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe\": container with ID starting with f56dc0f576847be44ad90b58a356d93f6503bb0351ca436f52bf78699c4b4fbe not found: ID does not exist" Nov 01 23:13:04.942969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.942933 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-envoy\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.942987 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-kube-api-access-vz4z8\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-kube-api-access-vz4z8\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943022 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4704702-163d-464d-b7fa-c0723a4ba01b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4704702-163d-464d-b7fa-c0723a4ba01b-istiod-ca-cert\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943054 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-podinfo\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943136 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-socket\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943163 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-credential-socket\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943191 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-data\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943259 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-certs\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943473 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-credential-socket" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.943531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943490 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-socket" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.943531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943510 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-data" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.943738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943543 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-token\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-token\") pod \"e4704702-163d-464d-b7fa-c0723a4ba01b\" (UID: \"e4704702-163d-464d-b7fa-c0723a4ba01b\") " Nov 01 23:13:04.943738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.943567 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4704702-163d-464d-b7fa-c0723a4ba01b-istiod-ca-cert" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:13:04.944432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.944384 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-certs" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.954281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.945553 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-kube-api-access-vz4z8" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "kube-api-access-vz4z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:13:04.954281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.946419 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-envoy" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:13:04.954487 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.954473 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4704702-163d-464d-b7fa-c0723a4ba01b-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:04.954686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.954496 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:04.954686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.954519 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:04.954686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.954534 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:04.954686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.954548 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:04.954937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.954716 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-podinfo" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:13:04.954937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:04.954794 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-token" pod "e4704702-163d-464d-b7fa-c0723a4ba01b" (UID: "e4704702-163d-464d-b7fa-c0723a4ba01b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:13:05.055826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:05.055791 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.055826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:05.055818 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.055826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:05.055830 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4z8\" (UniqueName: \"kubernetes.io/projected/e4704702-163d-464d-b7fa-c0723a4ba01b-kube-api-access-vz4z8\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.056065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:05.055841 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4704702-163d-464d-b7fa-c0723a4ba01b-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:13:05.235054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:05.234162 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj"] Nov 01 23:13:05.237542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:05.237516 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e5f47e-openshift-default-7d6b879557-tktsj"] Nov 01 23:13:06.733260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:06.733227 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" path="/var/lib/kubelet/pods/e4704702-163d-464d-b7fa-c0723a4ba01b/volumes" Nov 01 23:13:08.431785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:08.429076 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:08.527589 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:08.527540 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:09.741733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:09.741697 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:14.523663 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:13:14.523626 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:13:15.259380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:15.259348 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:15.611935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:15.611860 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:16.250614 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:16.250561 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:20.488895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.488861 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5"] Nov 01 23:13:20.489356 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489336 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489358 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3934c5-9084-4799-b0fe-9b6ac6a054db" containerName="istio-proxy" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489371 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489377 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489393 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489400 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489422 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489429 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489444 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f58e78c4-ffa9-4957-985d-682836f68224" containerName="payload-pre-processing" Nov 01 23:13:20.489449 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489452 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58e78c4-ffa9-4957-985d-682836f68224" containerName="payload-pre-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489470 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b153d778-544a-450d-8518-f1536e77e787" containerName="payload-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489476 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b153d778-544a-450d-8518-f1536e77e787" containerName="payload-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489500 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ebde4c11-a30e-48af-92bf-943ef2e40fdb" containerName="revoke-keys" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489507 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebde4c11-a30e-48af-92bf-943ef2e40fdb" containerName="revoke-keys" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489519 4981 cpu_manager.go:414] "containerMap: removing container" podUID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" containerName="payload-pre-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489527 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" containerName="payload-pre-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489537 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489542 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489654 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58e78c4-ffa9-4957-985d-682836f68224" containerName="payload-pre-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489674 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b153d778-544a-450d-8518-f1536e77e787" containerName="payload-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489689 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebde4c11-a30e-48af-92bf-943ef2e40fdb" containerName="revoke-keys" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489698 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="41289d4e-dbd9-425d-8f5e-e46cd1baa791" containerName="payload-pre-processing" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489706 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff04bd24-bb61-462b-acf7-1b11637a0eff" containerName="main" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489717 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4704702-163d-464d-b7fa-c0723a4ba01b" containerName="istio-proxy" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489726 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1794effc-dda4-4f3c-a529-5fa2fc655d6b" containerName="main" Nov 01 23:13:20.489872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.489736 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9775d38d-d819-4c0d-a5a4-5c006a9c2511" containerName="istio-proxy" Nov 01 23:13:20.494681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.494660 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.504667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.504639 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5"] Nov 01 23:13:20.592193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592146 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hsf\" (UniqueName: \"kubernetes.io/projected/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-kube-api-access-g4hsf\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592196 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592349 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592389 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592482 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592523 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592617 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592635 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.592707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.592683 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.693967 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694005 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694111 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694131 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694151 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694165 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694203 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694228 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hsf\" (UniqueName: \"kubernetes.io/projected/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-kube-api-access-g4hsf\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694266 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694256 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694429 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694527 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694608 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694897 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694757 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.694947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.694919 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.696546 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.696524 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.696874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.696857 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.701833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.701808 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.702214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.702194 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hsf\" (UniqueName: \"kubernetes.io/projected/4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7-kube-api-access-g4hsf\") pod \"maas-default-gateway-openshift-default-6886b96f9-7v6z5\" (UID: \"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:20.811250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:20.811168 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:21.090463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:21.090438 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5"] Nov 01 23:13:21.092474 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:13:21.092442 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2cb3a1_3ea5_436c_b204_c3f337bb2ec7.slice/crio-4bc200b77f4e25ba4aee3d9ab05376642a64fb16ee783543fe8ea4ee8107e7c8 WatchSource:0}: Error finding container 4bc200b77f4e25ba4aee3d9ab05376642a64fb16ee783543fe8ea4ee8107e7c8: Status 404 returned error can't find the container with id 4bc200b77f4e25ba4aee3d9ab05376642a64fb16ee783543fe8ea4ee8107e7c8 Nov 01 23:13:21.094742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:21.094712 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:13:21.094833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:21.094776 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:13:21.094833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:21.094805 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:13:22.030987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:22.030948 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" event={"ID":"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7","Type":"ContainerStarted","Data":"aec38adc58d7ec0852253d4d5a87fed7bd34251d832e28cbefd5973146aa3732"} Nov 01 23:13:22.031399 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:22.031008 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" event={"ID":"4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7","Type":"ContainerStarted","Data":"4bc200b77f4e25ba4aee3d9ab05376642a64fb16ee783543fe8ea4ee8107e7c8"} Nov 01 23:13:22.092598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:22.091926 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podStartSLOduration=2.091906251 podStartE2EDuration="2.091906251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:22.089187159 +0000 UTC m=+2967.517065092" watchObservedRunningTime="2026-11-01 23:13:22.091906251 +0000 UTC m=+2967.519784183" Nov 01 23:13:22.811588 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:22.811549 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:22.812930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:22.812896 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 23:13:22.813065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:22.812952 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 23:13:23.811842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:23.811805 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 23:13:23.812250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:23.811858 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 23:13:24.812085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:24.812051 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 23:13:24.812511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:24.812103 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 23:13:25.812235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:25.812189 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 23:13:25.812782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:25.812277 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 23:13:26.812527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:26.812489 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 23:13:26.812957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:26.812554 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 23:13:27.479847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:27.479813 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:27.811870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:27.811791 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 23:13:27.811870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:27.811843 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 23:13:29.811817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:29.811779 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:13:29.812283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:29.811835 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:13:30.852752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:30.852713 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:13:30.853246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:30.852774 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:13:31.230114 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:31.230084 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:31.230423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:31.230393 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:31.231229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:31.231211 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" Nov 01 23:13:32.904543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:32.904505 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:34.675799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:34.675761 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:35.503695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.503658 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8"] Nov 01 23:13:35.507920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.507892 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.526824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.526796 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8"] Nov 01 23:13:35.542304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542276 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542354 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542390 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5whjt\" (UniqueName: \"kubernetes.io/projected/3684469b-976a-4854-9697-a4d97edc5b16-kube-api-access-5whjt\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542433 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684469b-976a-4854-9697-a4d97edc5b16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542492 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542476 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542659 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684469b-976a-4854-9697-a4d97edc5b16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542695 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542689 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684469b-976a-4854-9697-a4d97edc5b16-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542706 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.542805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.542726 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643228 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643190 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643234 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5whjt\" (UniqueName: \"kubernetes.io/projected/3684469b-976a-4854-9697-a4d97edc5b16-kube-api-access-5whjt\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643274 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684469b-976a-4854-9697-a4d97edc5b16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643309 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643373 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684469b-976a-4854-9697-a4d97edc5b16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643402 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684469b-976a-4854-9697-a4d97edc5b16-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643424 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643452 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643479 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.643844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643755 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.644014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643870 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.644014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643895 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.644014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.643925 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.644163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.644147 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3684469b-976a-4854-9697-a4d97edc5b16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.645731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.645709 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3684469b-976a-4854-9697-a4d97edc5b16-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.645731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.645725 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3684469b-976a-4854-9697-a4d97edc5b16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.650975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.650943 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5whjt\" (UniqueName: \"kubernetes.io/projected/3684469b-976a-4854-9697-a4d97edc5b16-kube-api-access-5whjt\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.651103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.651084 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3684469b-976a-4854-9697-a4d97edc5b16-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6tll8\" (UID: \"3684469b-976a-4854-9697-a4d97edc5b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:35.828633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:35.828541 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:36.108201 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:36.108169 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8"] Nov 01 23:13:36.110033 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:13:36.110006 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3684469b_976a_4854_9697_a4d97edc5b16.slice/crio-d6466882302633e2e3b9fb079f9f5ae41bff014a3cf0ed2490f78255a486b1ef WatchSource:0}: Error finding container d6466882302633e2e3b9fb079f9f5ae41bff014a3cf0ed2490f78255a486b1ef: Status 404 returned error can't find the container with id d6466882302633e2e3b9fb079f9f5ae41bff014a3cf0ed2490f78255a486b1ef Nov 01 23:13:36.112311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:36.112273 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:13:36.112385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:36.112360 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:13:36.112425 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:36.112398 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:13:36.131252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:36.131221 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" event={"ID":"3684469b-976a-4854-9697-a4d97edc5b16","Type":"ContainerStarted","Data":"d6466882302633e2e3b9fb079f9f5ae41bff014a3cf0ed2490f78255a486b1ef"} Nov 01 23:13:37.139114 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:37.139076 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" event={"ID":"3684469b-976a-4854-9697-a4d97edc5b16","Type":"ContainerStarted","Data":"08ff516c0478058677afbb20401c2ca660b3a81ad8dffd826d8b3f75ed1e73df"} Nov 01 23:13:37.162121 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:37.162077 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podStartSLOduration=2.162064232 podStartE2EDuration="2.162064232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:13:37.15900591 +0000 UTC m=+2982.586883852" watchObservedRunningTime="2026-11-01 23:13:37.162064232 +0000 UTC m=+2982.589942162" Nov 01 23:13:37.829725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:37.829685 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:37.831319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:37.831296 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 01 23:13:37.831380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:37.831344 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 01 23:13:38.829848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:38.829804 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 01 23:13:38.830275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:38.829869 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 01 23:13:39.829046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:39.829015 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 01 23:13:39.829236 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:39.829062 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 01 23:13:40.829826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:40.829793 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 01 23:13:40.830225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:40.829853 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 01 23:13:41.829596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:41.829542 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 01 23:13:41.829797 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:41.829653 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 01 23:13:42.828855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:42.828825 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 01 23:13:42.829268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:42.828876 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 01 23:13:43.596140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:43.596108 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:13:44.071675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:44.071642 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:44.072204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:44.072065 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:44.073129 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:44.073109 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" Nov 01 23:13:44.523235 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:13:44.523204 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:13:54.027537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:54.027497 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88"] Nov 01 23:13:54.031541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:54.031516 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09ed389-9pr88"] Nov 01 23:13:54.732421 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:13:54.732388 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea8e03b-ddcb-4fc0-b09d-c9fb816315f0" path="/var/lib/kubelet/pods/bea8e03b-ddcb-4fc0-b09d-c9fb816315f0/volumes" Nov 01 23:14:03.529279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.529245 4981 scope.go:122] "RemoveContainer" containerID="6d723fc1f6de6a3e82b1728aae55fc3c0e8aa3770f6bd6782e2689e4d88803e9" Nov 01 23:14:03.541662 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.541639 4981 scope.go:122] "RemoveContainer" containerID="c37939650bb717fc70bf96b80d8fa514938d2e223e05e4c4d6beec493091fcda" Nov 01 23:14:03.550973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.550950 4981 scope.go:122] "RemoveContainer" containerID="33f64f78740dcfe5e8e9da231047c81f6d1ed22bf54c9765d4332da55fcad0ec" Nov 01 23:14:03.561174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.561153 4981 scope.go:122] "RemoveContainer" containerID="18b532a730863ed3c625480388f0be4a20c0f649a1d2737fe2adc2311e994279" Nov 01 23:14:03.570773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.570750 4981 scope.go:122] "RemoveContainer" containerID="253f575050f5c16a481d22c0ceed138b6d4bde16c4c0f7cab05aa25071c2f579" Nov 01 23:14:03.579948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.579923 4981 scope.go:122] "RemoveContainer" containerID="4e1bed05020a7f950790a3072b1209f9428e5e942ddac7deedd07184faa50fdd" Nov 01 23:14:03.589144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.589124 4981 scope.go:122] "RemoveContainer" containerID="2e5aef08d8d4d5e9a5c35b3dd86f7fd24f6d38e67ff625008e5e184b02b386bc" Nov 01 23:14:03.599193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:03.599170 4981 scope.go:122] "RemoveContainer" containerID="2a977c6af08e52d2a37d553ac2c9adf5012b04a9af3038eca9744bcee6c4753a" Nov 01 23:14:14.523339 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:14:14.523301 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:14:15.662907 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:15.662872 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:14:16.306516 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:16.306477 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:14:30.119599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:30.114688 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:30.119599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:30.114761 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:30.490598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:30.487700 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:14:30.490598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:30.487777 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:14:35.791349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:35.791310 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:14:44.523106 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:14:44.523069 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:14:44.857956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:44.857869 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57"] Nov 01 23:14:44.858194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:44.858114 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" podUID="1461c591-54fb-4cc8-a218-3e5bc6583857" containerName="discovery" containerID="cri-o://b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7" gracePeriod=30 Nov 01 23:14:45.023283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.023250 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx"] Nov 01 23:14:45.025692 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.025663 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d6ddc91-7zzjx"] Nov 01 23:14:45.027950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.027924 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z"] Nov 01 23:14:45.031778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.031757 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e0862ce-ssd2z"] Nov 01 23:14:45.123086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.123024 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 23:14:45.170269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.170230 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9"] Nov 01 23:14:45.170933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.170901 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1461c591-54fb-4cc8-a218-3e5bc6583857" containerName="discovery" Nov 01 23:14:45.170933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.170929 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1461c591-54fb-4cc8-a218-3e5bc6583857" containerName="discovery" Nov 01 23:14:45.171166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.171089 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1461c591-54fb-4cc8-a218-3e5bc6583857" containerName="discovery" Nov 01 23:14:45.174771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.174745 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.185226 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.185192 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9"] Nov 01 23:14:45.186732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.186251 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:14:45.186732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.186361 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:14:45.186732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.186380 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:14:45.186732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.186475 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:14:45.208110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.208085 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-kubeconfig\") pod \"1461c591-54fb-4cc8-a218-3e5bc6583857\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " Nov 01 23:14:45.208232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.208132 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1461c591-54fb-4cc8-a218-3e5bc6583857-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461c591-54fb-4cc8-a218-3e5bc6583857-local-certs\") pod \"1461c591-54fb-4cc8-a218-3e5bc6583857\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " Nov 01 23:14:45.208232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.208160 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-token\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-token\") pod \"1461c591-54fb-4cc8-a218-3e5bc6583857\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " Nov 01 23:14:45.208332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.208274 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-kube-api-access-twflh\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-kube-api-access-twflh\") pod \"1461c591-54fb-4cc8-a218-3e5bc6583857\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " Nov 01 23:14:45.208332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.208310 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-dns-cert\") pod \"1461c591-54fb-4cc8-a218-3e5bc6583857\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " Nov 01 23:14:45.208565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.208542 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-cacerts\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-cacerts\") pod \"1461c591-54fb-4cc8-a218-3e5bc6583857\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " Nov 01 23:14:45.208718 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.208699 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-ca-configmap\") pod \"1461c591-54fb-4cc8-a218-3e5bc6583857\" (UID: \"1461c591-54fb-4cc8-a218-3e5bc6583857\") " Nov 01 23:14:45.209056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.209031 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-ca-configmap" pod "1461c591-54fb-4cc8-a218-3e5bc6583857" (UID: "1461c591-54fb-4cc8-a218-3e5bc6583857"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:14:45.209291 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.209272 4981 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-ca-configmap\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:45.210792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.210763 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-token" pod "1461c591-54fb-4cc8-a218-3e5bc6583857" (UID: "1461c591-54fb-4cc8-a218-3e5bc6583857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:14:45.211003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.210974 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-dns-cert" pod "1461c591-54fb-4cc8-a218-3e5bc6583857" (UID: "1461c591-54fb-4cc8-a218-3e5bc6583857"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:14:45.211085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.210997 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1461c591-54fb-4cc8-a218-3e5bc6583857-local-certs" pod "1461c591-54fb-4cc8-a218-3e5bc6583857" (UID: "1461c591-54fb-4cc8-a218-3e5bc6583857"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:14:45.211154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.211073 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-kubeconfig" pod "1461c591-54fb-4cc8-a218-3e5bc6583857" (UID: "1461c591-54fb-4cc8-a218-3e5bc6583857"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:14:45.211276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.211247 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-kube-api-access-twflh" pod "1461c591-54fb-4cc8-a218-3e5bc6583857" (UID: "1461c591-54fb-4cc8-a218-3e5bc6583857"). InnerVolumeSpecName "kube-api-access-twflh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:14:45.211531 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.211514 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-cacerts" pod "1461c591-54fb-4cc8-a218-3e5bc6583857" (UID: "1461c591-54fb-4cc8-a218-3e5bc6583857"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:14:45.310454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310419 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.310675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310488 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.310675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310562 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpsk\" (UniqueName: \"kubernetes.io/projected/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-kube-api-access-kjpsk\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.310812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310709 4981 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-cacerts\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:45.310812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310731 4981 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-kubeconfig\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:45.310812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310742 4981 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1461c591-54fb-4cc8-a218-3e5bc6583857-local-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:45.310812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310751 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:45.310812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310761 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twflh\" (UniqueName: \"kubernetes.io/projected/1461c591-54fb-4cc8-a218-3e5bc6583857-kube-api-access-twflh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:45.310812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.310770 4981 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1461c591-54fb-4cc8-a218-3e5bc6583857-istio-csr-dns-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:45.411279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.411202 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.411279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.411265 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.411472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.411301 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpsk\" (UniqueName: \"kubernetes.io/projected/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-kube-api-access-kjpsk\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.412016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.411992 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.413427 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.413406 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.419659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.419638 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpsk\" (UniqueName: \"kubernetes.io/projected/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-kube-api-access-kjpsk\") pod \"maas-api-revoke-sub-ec843018-ggsv9\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.508341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.508292 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:45.637627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.637600 4981 generic.go:356] "Generic (PLEG): container finished" podID="1461c591-54fb-4cc8-a218-3e5bc6583857" containerID="b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7" exitCode=0 Nov 01 23:14:45.638102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.637708 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" Nov 01 23:14:45.638102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.637744 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" event={"ID":"1461c591-54fb-4cc8-a218-3e5bc6583857","Type":"ContainerDied","Data":"b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7"} Nov 01 23:14:45.638102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.637820 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57" event={"ID":"1461c591-54fb-4cc8-a218-3e5bc6583857","Type":"ContainerDied","Data":"5e652217b2bcdad7971610ae0f6d1c39cf42cef1836ec68753eee057cbefc9b7"} Nov 01 23:14:45.638102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.637854 4981 scope.go:122] "RemoveContainer" containerID="b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7" Nov 01 23:14:45.650988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.650966 4981 scope.go:122] "RemoveContainer" containerID="b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7" Nov 01 23:14:45.651254 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:14:45.651235 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7\": container with ID starting with b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7 not found: ID does not exist" containerID="b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7" Nov 01 23:14:45.651323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.651264 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7"} err="failed to get container status \"b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7\": rpc error: code = NotFound desc = could not find container \"b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7\": container with ID starting with b58abe882e624dc3c3dca193164380854db662a121c9da743096c853ba9decc7 not found: ID does not exist" Nov 01 23:14:45.662325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.662271 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57"] Nov 01 23:14:45.665375 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.665357 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b8x57"] Nov 01 23:14:45.787435 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:45.787407 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9"] Nov 01 23:14:45.788650 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:14:45.788622 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e466cd3_9a3a_43c3_84a2_1ccf81ca0a05.slice/crio-df62aaed9afe2c106026358af35b9edda62b5e852ce8f2ce03cc885f1d0f9a54 WatchSource:0}: Error finding container df62aaed9afe2c106026358af35b9edda62b5e852ce8f2ce03cc885f1d0f9a54: Status 404 returned error can't find the container with id df62aaed9afe2c106026358af35b9edda62b5e852ce8f2ce03cc885f1d0f9a54 Nov 01 23:14:46.646936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:46.646910 4981 generic.go:356] "Generic (PLEG): container finished" podID="6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" containerID="3fc43f7309514e4d4c5ad8ca0d42eb4014b7431486a7fb0bc85cafc2fbb5e492" exitCode=0 Nov 01 23:14:46.647311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:46.646955 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" event={"ID":"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05","Type":"ContainerDied","Data":"3fc43f7309514e4d4c5ad8ca0d42eb4014b7431486a7fb0bc85cafc2fbb5e492"} Nov 01 23:14:46.647311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:46.647007 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" event={"ID":"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05","Type":"ContainerStarted","Data":"df62aaed9afe2c106026358af35b9edda62b5e852ce8f2ce03cc885f1d0f9a54"} Nov 01 23:14:46.733289 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:46.733257 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1461c591-54fb-4cc8-a218-3e5bc6583857" path="/var/lib/kubelet/pods/1461c591-54fb-4cc8-a218-3e5bc6583857/volumes" Nov 01 23:14:46.733701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:46.733688 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80621398-2493-40fc-a370-001e18141226" path="/var/lib/kubelet/pods/80621398-2493-40fc-a370-001e18141226/volumes" Nov 01 23:14:46.734066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:46.734053 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7f5f55-b28d-4e63-8c30-36bb2ff39c99" path="/var/lib/kubelet/pods/bd7f5f55-b28d-4e63-8c30-36bb2ff39c99/volumes" Nov 01 23:14:47.795518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:47.795492 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:47.935650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:47.935619 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-xks-ca\") pod \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " Nov 01 23:14:47.935861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:47.935697 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-kube-api-access-kjpsk\" (UniqueName: \"kubernetes.io/projected/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-kube-api-access-kjpsk\") pod \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " Nov 01 23:14:47.935861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:47.935768 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-openshift-ca\") pod \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\" (UID: \"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05\") " Nov 01 23:14:47.936240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:47.936206 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-openshift-ca" pod "6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" (UID: "6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:14:47.938468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:47.938435 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-xks-ca" pod "6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" (UID: "6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:14:47.938468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:47.938454 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-kube-api-access-kjpsk" pod "6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" (UID: "6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05"). InnerVolumeSpecName "kube-api-access-kjpsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:14:48.037396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:48.037355 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:48.037595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:48.037400 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjpsk\" (UniqueName: \"kubernetes.io/projected/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-kube-api-access-kjpsk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:48.037595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:48.037418 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:14:48.662591 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:48.662544 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" event={"ID":"6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05","Type":"ContainerDied","Data":"df62aaed9afe2c106026358af35b9edda62b5e852ce8f2ce03cc885f1d0f9a54"} Nov 01 23:14:48.662591 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:48.662570 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9" Nov 01 23:14:48.662864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:14:48.662603 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df62aaed9afe2c106026358af35b9edda62b5e852ce8f2ce03cc885f1d0f9a54" Nov 01 23:15:00.133075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.133030 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7"] Nov 01 23:15:00.140537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.140504 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" containerName="revoke-keys" Nov 01 23:15:00.140774 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.140755 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" containerName="revoke-keys" Nov 01 23:15:00.141072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.141056 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" containerName="revoke-keys" Nov 01 23:15:00.144784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.144759 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.152372 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.152350 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:15:00.153214 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.153195 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:15:00.153481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.153464 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:15:00.156256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.156163 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892915-bxbt7\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.156365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.156286 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qzs\" (UniqueName: \"kubernetes.io/projected/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-kube-api-access-s6qzs\") pod \"maas-api-key-cleanup-29892915-bxbt7\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.158014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.157887 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7"] Nov 01 23:15:00.259357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.257881 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892915-bxbt7\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.259604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.259188 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892915-bxbt7\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.259760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.259714 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qzs\" (UniqueName: \"kubernetes.io/projected/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-kube-api-access-s6qzs\") pod \"maas-api-key-cleanup-29892915-bxbt7\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.269175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.269142 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qzs\" (UniqueName: \"kubernetes.io/projected/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-kube-api-access-s6qzs\") pod \"maas-api-key-cleanup-29892915-bxbt7\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.469310 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.469278 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:00.507598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.506686 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:15:00.507598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.506777 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:15:00.930225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:00.930195 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7"] Nov 01 23:15:01.763280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:01.763250 4981 generic.go:356] "Generic (PLEG): container finished" podID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerID="971ce10d13574b17d5c6642971e5297ffd135591f0b51d44fecb2113c409e67c" exitCode=60 Nov 01 23:15:01.763724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:01.763344 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" event={"ID":"1843d5a5-ff78-4cf4-8b72-38634fea2ec3","Type":"ContainerDied","Data":"971ce10d13574b17d5c6642971e5297ffd135591f0b51d44fecb2113c409e67c"} Nov 01 23:15:01.763724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:01.763404 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" event={"ID":"1843d5a5-ff78-4cf4-8b72-38634fea2ec3","Type":"ContainerStarted","Data":"597cbba730b53f2f74bbfe85a298688e5ab5dd222dbf37fbe4b6b323ee2c9e1a"} Nov 01 23:15:01.763724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:01.763596 4981 scope.go:122] "RemoveContainer" containerID="971ce10d13574b17d5c6642971e5297ffd135591f0b51d44fecb2113c409e67c" Nov 01 23:15:02.772033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:02.771994 4981 generic.go:356] "Generic (PLEG): container finished" podID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerID="7c8a5992d92ef9a4f29aab46cc7d8f9fccd048fa0e6f604dab40321a5715ae25" exitCode=60 Nov 01 23:15:02.772525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:02.772085 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" event={"ID":"1843d5a5-ff78-4cf4-8b72-38634fea2ec3","Type":"ContainerDied","Data":"7c8a5992d92ef9a4f29aab46cc7d8f9fccd048fa0e6f604dab40321a5715ae25"} Nov 01 23:15:02.772525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:02.772164 4981 scope.go:122] "RemoveContainer" containerID="971ce10d13574b17d5c6642971e5297ffd135591f0b51d44fecb2113c409e67c" Nov 01 23:15:02.772525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:02.772324 4981 scope.go:122] "RemoveContainer" containerID="7c8a5992d92ef9a4f29aab46cc7d8f9fccd048fa0e6f604dab40321a5715ae25" Nov 01 23:15:02.772694 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:15:02.772612 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892915-bxbt7_odh-ai-gateway-infra(1843d5a5-ff78-4cf4-8b72-38634fea2ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" Nov 01 23:15:03.665276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:03.665240 4981 scope.go:122] "RemoveContainer" containerID="0900a78318e6e5946b3ed08c46eaa3aecc2c99e3b3e42f9b3d366a7df8de57f4" Nov 01 23:15:03.674655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:03.674635 4981 scope.go:122] "RemoveContainer" containerID="75958c17b008925aa82b111c334031ec4db5da3c3e88412ea051dd75ab63ed62" Nov 01 23:15:03.683564 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:03.683538 4981 scope.go:122] "RemoveContainer" containerID="1e8b3df0081394e7f2a1eb258de1a72c6c74f2cf8fb5472854b2b9afa2f2feb1" Nov 01 23:15:03.692174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:03.692152 4981 scope.go:122] "RemoveContainer" containerID="21b5826ee32d4c5472fb31414d10a34ea07c1f9ff0215aeaa95432049e108d78" Nov 01 23:15:03.700733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:03.700710 4981 scope.go:122] "RemoveContainer" containerID="74e36e22ffdfaf73514ac479a4e549c5fa991ca6ee9794abbe7e43c0eb7aaed3" Nov 01 23:15:03.709650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:03.709629 4981 scope.go:122] "RemoveContainer" containerID="e378f8f523f97f91014d4627fbcbba1fc4b8d27bfee3aee0991bc0719520b94b" Nov 01 23:15:03.779888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:03.779861 4981 scope.go:122] "RemoveContainer" containerID="7c8a5992d92ef9a4f29aab46cc7d8f9fccd048fa0e6f604dab40321a5715ae25" Nov 01 23:15:03.780260 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:15:03.780106 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892915-bxbt7_odh-ai-gateway-infra(1843d5a5-ff78-4cf4-8b72-38634fea2ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" Nov 01 23:15:14.523507 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:15:14.523468 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:15:15.697511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:15.697473 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:15:15.727326 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:15.727298 4981 scope.go:122] "RemoveContainer" containerID="7c8a5992d92ef9a4f29aab46cc7d8f9fccd048fa0e6f604dab40321a5715ae25" Nov 01 23:15:16.337734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:16.337658 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:15:16.863920 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:16.863886 4981 generic.go:356] "Generic (PLEG): container finished" podID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerID="56bd277f198b3fa3a26699de65aaee73718c0552a0e8d9ce1ddef78ae01eaeee" exitCode=60 Nov 01 23:15:16.864382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:16.863951 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" event={"ID":"1843d5a5-ff78-4cf4-8b72-38634fea2ec3","Type":"ContainerDied","Data":"56bd277f198b3fa3a26699de65aaee73718c0552a0e8d9ce1ddef78ae01eaeee"} Nov 01 23:15:16.864382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:16.864016 4981 scope.go:122] "RemoveContainer" containerID="7c8a5992d92ef9a4f29aab46cc7d8f9fccd048fa0e6f604dab40321a5715ae25" Nov 01 23:15:16.864382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:16.864358 4981 scope.go:122] "RemoveContainer" containerID="56bd277f198b3fa3a26699de65aaee73718c0552a0e8d9ce1ddef78ae01eaeee" Nov 01 23:15:16.864631 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:15:16.864611 4981 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892915-bxbt7_odh-ai-gateway-infra(1843d5a5-ff78-4cf4-8b72-38634fea2ec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" Nov 01 23:15:17.906792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:17.906754 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7"] Nov 01 23:15:18.048067 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.048039 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:18.140678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.140643 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-maas-api-service-ca\") pod \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " Nov 01 23:15:18.140861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.140701 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-kube-api-access-s6qzs\" (UniqueName: \"kubernetes.io/projected/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-kube-api-access-s6qzs\") pod \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\" (UID: \"1843d5a5-ff78-4cf4-8b72-38634fea2ec3\") " Nov 01 23:15:18.141104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.141076 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-maas-api-service-ca" pod "1843d5a5-ff78-4cf4-8b72-38634fea2ec3" (UID: "1843d5a5-ff78-4cf4-8b72-38634fea2ec3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:15:18.142781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.142754 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-kube-api-access-s6qzs" pod "1843d5a5-ff78-4cf4-8b72-38634fea2ec3" (UID: "1843d5a5-ff78-4cf4-8b72-38634fea2ec3"). InnerVolumeSpecName "kube-api-access-s6qzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:15:18.241783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.241751 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6qzs\" (UniqueName: \"kubernetes.io/projected/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-kube-api-access-s6qzs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:18.241783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.241780 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1843d5a5-ff78-4cf4-8b72-38634fea2ec3-maas-api-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:18.879860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.879831 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" Nov 01 23:15:18.880124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.879864 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7" event={"ID":"1843d5a5-ff78-4cf4-8b72-38634fea2ec3","Type":"ContainerDied","Data":"597cbba730b53f2f74bbfe85a298688e5ab5dd222dbf37fbe4b6b323ee2c9e1a"} Nov 01 23:15:18.880124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.879931 4981 scope.go:122] "RemoveContainer" containerID="56bd277f198b3fa3a26699de65aaee73718c0552a0e8d9ce1ddef78ae01eaeee" Nov 01 23:15:18.899111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.899086 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7"] Nov 01 23:15:18.904480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:18.904460 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892915-bxbt7"] Nov 01 23:15:20.731899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:20.731856 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" path="/var/lib/kubelet/pods/1843d5a5-ff78-4cf4-8b72-38634fea2ec3/volumes" Nov 01 23:15:21.641290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.641251 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6"] Nov 01 23:15:21.641745 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.641730 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:21.641795 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.641746 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:21.641795 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.641777 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:21.641795 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.641783 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:21.641919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.641857 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:21.641919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.641867 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:21.646652 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.646635 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.650686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.650665 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:15:21.651898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.651865 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:15:21.652658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.652631 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6"] Nov 01 23:15:21.691481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.691453 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:15:21.691674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.691563 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:15:21.772809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.772776 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5f7g\" (UniqueName: \"kubernetes.io/projected/f1b83401-d56c-47fe-b856-43187a10d73b-kube-api-access-g5f7g\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.773177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.772820 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.773177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.772901 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.873951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.873902 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5f7g\" (UniqueName: \"kubernetes.io/projected/f1b83401-d56c-47fe-b856-43187a10d73b-kube-api-access-g5f7g\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.873951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.873961 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.874164 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.873986 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.874606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.874588 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.876365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.876345 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.882215 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.882191 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5f7g\" (UniqueName: \"kubernetes.io/projected/f1b83401-d56c-47fe-b856-43187a10d73b-kube-api-access-g5f7g\") pod \"maas-api-revoke-sub-d3fba056-27kz6\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:21.962817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:21.962791 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:22.238552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:22.238527 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6"] Nov 01 23:15:22.240470 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:15:22.240440 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b83401_d56c_47fe_b856_43187a10d73b.slice/crio-fc779fb608a8cc2d6a2cc7dc2ffc8fe66a12f0b97ee832acda39214ec7eaaf51 WatchSource:0}: Error finding container fc779fb608a8cc2d6a2cc7dc2ffc8fe66a12f0b97ee832acda39214ec7eaaf51: Status 404 returned error can't find the container with id fc779fb608a8cc2d6a2cc7dc2ffc8fe66a12f0b97ee832acda39214ec7eaaf51 Nov 01 23:15:22.913657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:22.913523 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" event={"ID":"f1b83401-d56c-47fe-b856-43187a10d73b","Type":"ContainerStarted","Data":"93e04419310b8ad071cd37b5005996b2ccbd52651ea481d776272774ee00d9b7"} Nov 01 23:15:22.913657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:22.913626 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" event={"ID":"f1b83401-d56c-47fe-b856-43187a10d73b","Type":"ContainerStarted","Data":"fc779fb608a8cc2d6a2cc7dc2ffc8fe66a12f0b97ee832acda39214ec7eaaf51"} Nov 01 23:15:22.930113 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:22.930061 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" podStartSLOduration=1.930046857 podStartE2EDuration="1.930046857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:15:22.929395035 +0000 UTC m=+3088.357272967" watchObservedRunningTime="2026-11-01 23:15:22.930046857 +0000 UTC m=+3088.357924789" Nov 01 23:15:23.951359 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:23.951317 4981 generic.go:356] "Generic (PLEG): container finished" podID="f1b83401-d56c-47fe-b856-43187a10d73b" containerID="93e04419310b8ad071cd37b5005996b2ccbd52651ea481d776272774ee00d9b7" exitCode=0 Nov 01 23:15:23.951898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:23.951638 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" event={"ID":"f1b83401-d56c-47fe-b856-43187a10d73b","Type":"ContainerDied","Data":"93e04419310b8ad071cd37b5005996b2ccbd52651ea481d776272774ee00d9b7"} Nov 01 23:15:25.226705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.226646 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:25.339963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.339933 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-openshift-ca\") pod \"f1b83401-d56c-47fe-b856-43187a10d73b\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " Nov 01 23:15:25.340155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.340000 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b83401-d56c-47fe-b856-43187a10d73b-kube-api-access-g5f7g\" (UniqueName: \"kubernetes.io/projected/f1b83401-d56c-47fe-b856-43187a10d73b-kube-api-access-g5f7g\") pod \"f1b83401-d56c-47fe-b856-43187a10d73b\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " Nov 01 23:15:25.340155 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.340052 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-xks-ca\") pod \"f1b83401-d56c-47fe-b856-43187a10d73b\" (UID: \"f1b83401-d56c-47fe-b856-43187a10d73b\") " Nov 01 23:15:25.340334 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.340310 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-openshift-ca" pod "f1b83401-d56c-47fe-b856-43187a10d73b" (UID: "f1b83401-d56c-47fe-b856-43187a10d73b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:15:25.342144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.342112 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-xks-ca" pod "f1b83401-d56c-47fe-b856-43187a10d73b" (UID: "f1b83401-d56c-47fe-b856-43187a10d73b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:15:25.342257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.342165 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b83401-d56c-47fe-b856-43187a10d73b-kube-api-access-g5f7g" pod "f1b83401-d56c-47fe-b856-43187a10d73b" (UID: "f1b83401-d56c-47fe-b856-43187a10d73b"). InnerVolumeSpecName "kube-api-access-g5f7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:15:25.441975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.441942 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:25.441975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.441968 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5f7g\" (UniqueName: \"kubernetes.io/projected/f1b83401-d56c-47fe-b856-43187a10d73b-kube-api-access-g5f7g\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:25.441975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.441979 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1b83401-d56c-47fe-b856-43187a10d73b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:25.967418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.967384 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" Nov 01 23:15:25.967645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.967417 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6" event={"ID":"f1b83401-d56c-47fe-b856-43187a10d73b","Type":"ContainerDied","Data":"fc779fb608a8cc2d6a2cc7dc2ffc8fe66a12f0b97ee832acda39214ec7eaaf51"} Nov 01 23:15:25.967645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:25.967465 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc779fb608a8cc2d6a2cc7dc2ffc8fe66a12f0b97ee832acda39214ec7eaaf51" Nov 01 23:15:32.020474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:32.020440 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj"] Nov 01 23:15:32.024686 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:32.024663 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f749f3a1-dq7mj"] Nov 01 23:15:32.733887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:32.733851 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1f7468-8561-40a0-ae39-70aafe1a7cb8" path="/var/lib/kubelet/pods/cc1f7468-8561-40a0-ae39-70aafe1a7cb8/volumes" Nov 01 23:15:33.020337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:33.020258 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf"] Nov 01 23:15:33.024380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:33.024355 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-376015f8-pxnsf"] Nov 01 23:15:34.739922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:34.739891 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da815bca-be05-447f-80a2-90ace52e4e04" path="/var/lib/kubelet/pods/da815bca-be05-447f-80a2-90ace52e4e04/volumes" Nov 01 23:15:35.021314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:35.021216 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb"] Nov 01 23:15:35.024380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:35.024355 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d6eed9c-sf4fb"] Nov 01 23:15:36.734243 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:36.734207 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3baea25-dcda-4716-9176-538077fedfa1" path="/var/lib/kubelet/pods/a3baea25-dcda-4716-9176-538077fedfa1/volumes" Nov 01 23:15:41.470079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.470047 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s"] Nov 01 23:15:41.470544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.470525 4981 cpu_manager.go:414] "containerMap: removing container" podUID="f1b83401-d56c-47fe-b856-43187a10d73b" containerName="revoke-keys" Nov 01 23:15:41.470544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.470539 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b83401-d56c-47fe-b856-43187a10d73b" containerName="revoke-keys" Nov 01 23:15:41.470659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.470565 4981 cpu_manager.go:414] "containerMap: removing container" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:41.470659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.470583 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:41.470659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.470651 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b83401-d56c-47fe-b856-43187a10d73b" containerName="revoke-keys" Nov 01 23:15:41.470777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.470661 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="1843d5a5-ff78-4cf4-8b72-38634fea2ec3" containerName="cleanup" Nov 01 23:15:41.475149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.475133 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.479848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.479823 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:15:41.480015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.479995 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:15:41.480077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.479951 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:15:41.480077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.480069 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:15:41.481544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.481511 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s"] Nov 01 23:15:41.492453 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.492428 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxwk\" (UniqueName: \"kubernetes.io/projected/53b87954-c27e-46d2-9ad2-7ed126e5f993-kube-api-access-bnxwk\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.492566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.492483 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.492566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.492505 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.593329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.593298 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.593329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.593340 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.593548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.593406 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxwk\" (UniqueName: \"kubernetes.io/projected/53b87954-c27e-46d2-9ad2-7ed126e5f993-kube-api-access-bnxwk\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.593980 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.593956 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.595730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.595708 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.601403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.601384 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxwk\" (UniqueName: \"kubernetes.io/projected/53b87954-c27e-46d2-9ad2-7ed126e5f993-kube-api-access-bnxwk\") pod \"maas-api-revoke-sub-ae594979-gw74s\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:41.791041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:41.790953 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:42.063930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:42.063902 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s"] Nov 01 23:15:42.065697 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:15:42.065670 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b87954_c27e_46d2_9ad2_7ed126e5f993.slice/crio-0b83726effd9c44cee60228e3e616e1ea3d329308b243528af1f7dc4b1917446 WatchSource:0}: Error finding container 0b83726effd9c44cee60228e3e616e1ea3d329308b243528af1f7dc4b1917446: Status 404 returned error can't find the container with id 0b83726effd9c44cee60228e3e616e1ea3d329308b243528af1f7dc4b1917446 Nov 01 23:15:42.084638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:42.084609 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" event={"ID":"53b87954-c27e-46d2-9ad2-7ed126e5f993","Type":"ContainerStarted","Data":"0b83726effd9c44cee60228e3e616e1ea3d329308b243528af1f7dc4b1917446"} Nov 01 23:15:43.092249 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:43.092155 4981 generic.go:356] "Generic (PLEG): container finished" podID="53b87954-c27e-46d2-9ad2-7ed126e5f993" containerID="f44b340a25db84c9939efef81603763aa8ce2662697544a8058506eab7b16140" exitCode=0 Nov 01 23:15:43.092707 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:43.092244 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" event={"ID":"53b87954-c27e-46d2-9ad2-7ed126e5f993","Type":"ContainerDied","Data":"f44b340a25db84c9939efef81603763aa8ce2662697544a8058506eab7b16140"} Nov 01 23:15:44.233265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.233238 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:44.319335 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.319306 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-xks-ca\") pod \"53b87954-c27e-46d2-9ad2-7ed126e5f993\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " Nov 01 23:15:44.319544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.319390 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b87954-c27e-46d2-9ad2-7ed126e5f993-kube-api-access-bnxwk\" (UniqueName: \"kubernetes.io/projected/53b87954-c27e-46d2-9ad2-7ed126e5f993-kube-api-access-bnxwk\") pod \"53b87954-c27e-46d2-9ad2-7ed126e5f993\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " Nov 01 23:15:44.319544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.319508 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-openshift-ca\") pod \"53b87954-c27e-46d2-9ad2-7ed126e5f993\" (UID: \"53b87954-c27e-46d2-9ad2-7ed126e5f993\") " Nov 01 23:15:44.319886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.319864 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-openshift-ca" pod "53b87954-c27e-46d2-9ad2-7ed126e5f993" (UID: "53b87954-c27e-46d2-9ad2-7ed126e5f993"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:15:44.321374 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.321353 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-xks-ca" pod "53b87954-c27e-46d2-9ad2-7ed126e5f993" (UID: "53b87954-c27e-46d2-9ad2-7ed126e5f993"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:15:44.321585 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.321557 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b87954-c27e-46d2-9ad2-7ed126e5f993-kube-api-access-bnxwk" pod "53b87954-c27e-46d2-9ad2-7ed126e5f993" (UID: "53b87954-c27e-46d2-9ad2-7ed126e5f993"). InnerVolumeSpecName "kube-api-access-bnxwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:15:44.421006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.420924 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:44.421006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.420956 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxwk\" (UniqueName: \"kubernetes.io/projected/53b87954-c27e-46d2-9ad2-7ed126e5f993-kube-api-access-bnxwk\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:44.421006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:44.420970 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b87954-c27e-46d2-9ad2-7ed126e5f993-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:15:44.522534 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:15:44.522494 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:15:45.020152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:45.020056 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl"] Nov 01 23:15:45.022227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:45.022198 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-419707d8-48mcl"] Nov 01 23:15:45.107518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:45.107492 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" Nov 01 23:15:45.107729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:45.107515 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s" event={"ID":"53b87954-c27e-46d2-9ad2-7ed126e5f993","Type":"ContainerDied","Data":"0b83726effd9c44cee60228e3e616e1ea3d329308b243528af1f7dc4b1917446"} Nov 01 23:15:45.107729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:45.107570 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b83726effd9c44cee60228e3e616e1ea3d329308b243528af1f7dc4b1917446" Nov 01 23:15:46.732489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:46.732456 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77580d9e-a603-4a05-b853-1df49f18d2c2" path="/var/lib/kubelet/pods/77580d9e-a603-4a05-b853-1df49f18d2c2/volumes" Nov 01 23:15:55.022527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:55.022477 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9"] Nov 01 23:15:55.026012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:55.025981 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72f93753-29ds9"] Nov 01 23:15:56.744607 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:56.743421 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65153a3e-b028-4d7c-9285-cefad05006a5" path="/var/lib/kubelet/pods/65153a3e-b028-4d7c-9285-cefad05006a5/volumes" Nov 01 23:15:59.378052 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.378016 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz"] Nov 01 23:15:59.378443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.378426 4981 cpu_manager.go:414] "containerMap: removing container" podUID="53b87954-c27e-46d2-9ad2-7ed126e5f993" containerName="revoke-keys" Nov 01 23:15:59.378443 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.378436 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b87954-c27e-46d2-9ad2-7ed126e5f993" containerName="revoke-keys" Nov 01 23:15:59.378528 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.378520 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b87954-c27e-46d2-9ad2-7ed126e5f993" containerName="revoke-keys" Nov 01 23:15:59.381976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.381956 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.390502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.389738 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz"] Nov 01 23:15:59.405180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.405156 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:15:59.405307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.405255 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:15:59.450119 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.450090 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:15:59.450235 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.450117 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:15:59.483894 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.483868 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.484000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.483915 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82ch5\" (UniqueName: \"kubernetes.io/projected/fc104d44-8de8-4737-9016-48ea845edee5-kube-api-access-82ch5\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.484074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.484054 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.584930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.584893 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.585103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.584944 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82ch5\" (UniqueName: \"kubernetes.io/projected/fc104d44-8de8-4737-9016-48ea845edee5-kube-api-access-82ch5\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.585103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.585003 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.585709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.585686 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.587333 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.587305 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.592457 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.592433 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82ch5\" (UniqueName: \"kubernetes.io/projected/fc104d44-8de8-4737-9016-48ea845edee5-kube-api-access-82ch5\") pod \"maas-api-revoke-sub-a4aef7d7-t6wwz\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.696466 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.696436 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:15:59.970847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:15:59.970815 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz"] Nov 01 23:15:59.971985 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:15:59.971952 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc104d44_8de8_4737_9016_48ea845edee5.slice/crio-e8272a52735ca166b962dd715f9274ffe99a54a1e57e10a03984bc775767f60e WatchSource:0}: Error finding container e8272a52735ca166b962dd715f9274ffe99a54a1e57e10a03984bc775767f60e: Status 404 returned error can't find the container with id e8272a52735ca166b962dd715f9274ffe99a54a1e57e10a03984bc775767f60e Nov 01 23:16:00.220736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:00.220657 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" event={"ID":"fc104d44-8de8-4737-9016-48ea845edee5","Type":"ContainerStarted","Data":"e8272a52735ca166b962dd715f9274ffe99a54a1e57e10a03984bc775767f60e"} Nov 01 23:16:01.228065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:01.228029 4981 generic.go:356] "Generic (PLEG): container finished" podID="fc104d44-8de8-4737-9016-48ea845edee5" containerID="06655feadd602a2d4d521a8fb25d41455cdde8d3d8f59485d0e36424ec8c45b5" exitCode=0 Nov 01 23:16:01.228486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:01.228126 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" event={"ID":"fc104d44-8de8-4737-9016-48ea845edee5","Type":"ContainerDied","Data":"06655feadd602a2d4d521a8fb25d41455cdde8d3d8f59485d0e36424ec8c45b5"} Nov 01 23:16:02.372402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.372374 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:16:02.412852 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.412800 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-openshift-ca\") pod \"fc104d44-8de8-4737-9016-48ea845edee5\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " Nov 01 23:16:02.413039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.412920 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc104d44-8de8-4737-9016-48ea845edee5-kube-api-access-82ch5\" (UniqueName: \"kubernetes.io/projected/fc104d44-8de8-4737-9016-48ea845edee5-kube-api-access-82ch5\") pod \"fc104d44-8de8-4737-9016-48ea845edee5\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " Nov 01 23:16:02.413039 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.412953 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-xks-ca\") pod \"fc104d44-8de8-4737-9016-48ea845edee5\" (UID: \"fc104d44-8de8-4737-9016-48ea845edee5\") " Nov 01 23:16:02.413238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.413215 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-openshift-ca" pod "fc104d44-8de8-4737-9016-48ea845edee5" (UID: "fc104d44-8de8-4737-9016-48ea845edee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:16:02.413330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.413298 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:02.415125 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.415103 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-xks-ca" pod "fc104d44-8de8-4737-9016-48ea845edee5" (UID: "fc104d44-8de8-4737-9016-48ea845edee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:16:02.415622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.415590 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc104d44-8de8-4737-9016-48ea845edee5-kube-api-access-82ch5" pod "fc104d44-8de8-4737-9016-48ea845edee5" (UID: "fc104d44-8de8-4737-9016-48ea845edee5"). InnerVolumeSpecName "kube-api-access-82ch5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:16:02.514222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.514135 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82ch5\" (UniqueName: \"kubernetes.io/projected/fc104d44-8de8-4737-9016-48ea845edee5-kube-api-access-82ch5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:02.514222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:02.514169 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc104d44-8de8-4737-9016-48ea845edee5-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:03.241955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.241926 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" Nov 01 23:16:03.242246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.241919 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz" event={"ID":"fc104d44-8de8-4737-9016-48ea845edee5","Type":"ContainerDied","Data":"e8272a52735ca166b962dd715f9274ffe99a54a1e57e10a03984bc775767f60e"} Nov 01 23:16:03.242246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.242051 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8272a52735ca166b962dd715f9274ffe99a54a1e57e10a03984bc775767f60e" Nov 01 23:16:03.760777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.760743 4981 scope.go:122] "RemoveContainer" containerID="63d0196cb68bc6b9a7654dbc82996c33918de8682b51f2f2aca000e3a1ab3e92" Nov 01 23:16:03.770517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.770494 4981 scope.go:122] "RemoveContainer" containerID="92952219c6f87c63822cf937302b5e7912cefaa4eb12be557a6d2b7be0f5c884" Nov 01 23:16:03.779786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.779765 4981 scope.go:122] "RemoveContainer" containerID="51efe787ebf57f69afc60dd8b0a5d84fe86a0588ac00aa32bd189a11486ae1e2" Nov 01 23:16:03.788281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.788259 4981 scope.go:122] "RemoveContainer" containerID="fc746415e9feaf6ab85780d33832c9cd93e06b95c6e50ab603026f14303d0257" Nov 01 23:16:03.797192 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:03.797175 4981 scope.go:122] "RemoveContainer" containerID="5ef9b45b0a61eb9cf0627c0d9cd4f4a4e30f8b8aa14db5427185dc7c3b9a2058" Nov 01 23:16:05.024713 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:05.024611 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk"] Nov 01 23:16:05.031263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:05.026962 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a595d015-x74qk"] Nov 01 23:16:06.732257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:06.732223 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39948769-7fef-4a5c-aedd-b67ec3528417" path="/var/lib/kubelet/pods/39948769-7fef-4a5c-aedd-b67ec3528417/volumes" Nov 01 23:16:14.568708 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:16:14.568669 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:16:15.120600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:15.119705 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:16:15.120600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:15.119811 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:16:17.194525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.194494 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:16:17.295537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.295502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:16:17.963411 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.963376 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b"] Nov 01 23:16:17.963814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.963801 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fc104d44-8de8-4737-9016-48ea845edee5" containerName="revoke-keys" Nov 01 23:16:17.963876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.963816 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc104d44-8de8-4737-9016-48ea845edee5" containerName="revoke-keys" Nov 01 23:16:17.963916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.963902 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc104d44-8de8-4737-9016-48ea845edee5" containerName="revoke-keys" Nov 01 23:16:17.969135 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.969119 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:17.973084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.972839 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:16:17.973084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.972889 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:16:17.974009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.973985 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:16:17.974721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.974060 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:16:17.977260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:17.977236 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b"] Nov 01 23:16:18.086054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.086013 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.086054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.086057 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.086290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.086171 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76r5\" (UniqueName: \"kubernetes.io/projected/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-kube-api-access-w76r5\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.186904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.186868 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.187111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.186919 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.187111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.186992 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76r5\" (UniqueName: \"kubernetes.io/projected/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-kube-api-access-w76r5\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.187666 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.187645 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.189311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.189288 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.195669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.195647 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76r5\" (UniqueName: \"kubernetes.io/projected/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-kube-api-access-w76r5\") pod \"maas-api-revoke-sub-79805ca9-xm89b\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.283817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.283723 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:18.560744 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:18.560719 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b"] Nov 01 23:16:18.563468 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:16:18.563438 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24427c8d_16ac_4bd2_8a4b_b5a0c822ac7f.slice/crio-1e008339b8107e42b7d1f2660c60707b52c05d9fdf6972200df6daf65f05ef38 WatchSource:0}: Error finding container 1e008339b8107e42b7d1f2660c60707b52c05d9fdf6972200df6daf65f05ef38: Status 404 returned error can't find the container with id 1e008339b8107e42b7d1f2660c60707b52c05d9fdf6972200df6daf65f05ef38 Nov 01 23:16:19.370621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:19.370569 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" event={"ID":"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f","Type":"ContainerStarted","Data":"24f0c5886a37d5aa83881ee0734e8c6230be19eac39ac75469f96438286628f8"} Nov 01 23:16:19.371043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:19.370672 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" event={"ID":"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f","Type":"ContainerStarted","Data":"1e008339b8107e42b7d1f2660c60707b52c05d9fdf6972200df6daf65f05ef38"} Nov 01 23:16:19.391506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:19.391446 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" podStartSLOduration=2.391427503 podStartE2EDuration="2.391427503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:16:19.388625277 +0000 UTC m=+3144.816503209" watchObservedRunningTime="2026-11-01 23:16:19.391427503 +0000 UTC m=+3144.819305435" Nov 01 23:16:20.378361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:20.378324 4981 generic.go:356] "Generic (PLEG): container finished" podID="24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" containerID="24f0c5886a37d5aa83881ee0734e8c6230be19eac39ac75469f96438286628f8" exitCode=0 Nov 01 23:16:20.378792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:20.378415 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" event={"ID":"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f","Type":"ContainerDied","Data":"24f0c5886a37d5aa83881ee0734e8c6230be19eac39ac75469f96438286628f8"} Nov 01 23:16:21.513941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.513917 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:21.619386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.619341 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-xks-ca\") pod \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " Nov 01 23:16:21.619627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.619422 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-openshift-ca\") pod \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " Nov 01 23:16:21.619627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.619522 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-kube-api-access-w76r5\" (UniqueName: \"kubernetes.io/projected/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-kube-api-access-w76r5\") pod \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\" (UID: \"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f\") " Nov 01 23:16:21.619818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.619787 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-openshift-ca" pod "24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" (UID: "24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:16:21.621535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.621513 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-xks-ca" pod "24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" (UID: "24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:16:21.621872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.621855 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-kube-api-access-w76r5" pod "24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" (UID: "24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f"). InnerVolumeSpecName "kube-api-access-w76r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:16:21.721104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.721072 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:21.721104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.721099 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w76r5\" (UniqueName: \"kubernetes.io/projected/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-kube-api-access-w76r5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:21.721307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:21.721111 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:22.391908 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:22.391879 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" Nov 01 23:16:22.392102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:22.391917 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b" event={"ID":"24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f","Type":"ContainerDied","Data":"1e008339b8107e42b7d1f2660c60707b52c05d9fdf6972200df6daf65f05ef38"} Nov 01 23:16:22.392102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:22.391964 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e008339b8107e42b7d1f2660c60707b52c05d9fdf6972200df6daf65f05ef38" Nov 01 23:16:23.022159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:23.022119 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7"] Nov 01 23:16:23.024539 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:23.024511 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b43b1ab-m5kv7"] Nov 01 23:16:24.732801 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:24.732766 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c4903d-3833-40e5-88ee-d0369d2d4092" path="/var/lib/kubelet/pods/f8c4903d-3833-40e5-88ee-d0369d2d4092/volumes" Nov 01 23:16:33.033702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:33.033655 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l"] Nov 01 23:16:33.037807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:33.037784 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75c6ea04-vgx8l"] Nov 01 23:16:34.732476 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:34.732441 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36da46bf-f910-44ac-8dbf-f966a4a61798" path="/var/lib/kubelet/pods/36da46bf-f910-44ac-8dbf-f966a4a61798/volumes" Nov 01 23:16:35.644782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.644747 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g"] Nov 01 23:16:35.645172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.645159 4981 cpu_manager.go:414] "containerMap: removing container" podUID="24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" containerName="revoke-keys" Nov 01 23:16:35.645229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.645173 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" containerName="revoke-keys" Nov 01 23:16:35.645276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.645249 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" containerName="revoke-keys" Nov 01 23:16:35.648456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.648439 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.655096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.655073 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:16:35.655336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.655319 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:16:35.655482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.655351 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:16:35.655482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.655234 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:16:35.657148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.657124 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g"] Nov 01 23:16:35.771283 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.771241 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.771688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.771294 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.771688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.771372 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjgn\" (UniqueName: \"kubernetes.io/projected/b1f5a86e-8126-402f-90dc-e64752437f8e-kube-api-access-qbjgn\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.871955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.871923 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjgn\" (UniqueName: \"kubernetes.io/projected/b1f5a86e-8126-402f-90dc-e64752437f8e-kube-api-access-qbjgn\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.872102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.872009 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.872102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.872054 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.872738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.872713 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.874318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.874291 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.880357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.880334 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjgn\" (UniqueName: \"kubernetes.io/projected/b1f5a86e-8126-402f-90dc-e64752437f8e-kube-api-access-qbjgn\") pod \"maas-api-revoke-sub-0b7e8edb-7296g\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:35.964066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:35.964037 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:36.236997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:36.236970 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g"] Nov 01 23:16:36.239427 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:16:36.239400 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f5a86e_8126_402f_90dc_e64752437f8e.slice/crio-b51cbefee5f328b74f0bdbfdff9469b04de84f737967db07eb2180c635f1f72f WatchSource:0}: Error finding container b51cbefee5f328b74f0bdbfdff9469b04de84f737967db07eb2180c635f1f72f: Status 404 returned error can't find the container with id b51cbefee5f328b74f0bdbfdff9469b04de84f737967db07eb2180c635f1f72f Nov 01 23:16:36.486455 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:36.486385 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" event={"ID":"b1f5a86e-8126-402f-90dc-e64752437f8e","Type":"ContainerStarted","Data":"b51cbefee5f328b74f0bdbfdff9469b04de84f737967db07eb2180c635f1f72f"} Nov 01 23:16:37.494662 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:37.494623 4981 generic.go:356] "Generic (PLEG): container finished" podID="b1f5a86e-8126-402f-90dc-e64752437f8e" containerID="bdc19c387eeebf3635e23eaf8cabcffe285ecc435ec2eeddd09fd4ae214ccc7c" exitCode=0 Nov 01 23:16:37.495098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:37.494716 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" event={"ID":"b1f5a86e-8126-402f-90dc-e64752437f8e","Type":"ContainerDied","Data":"bdc19c387eeebf3635e23eaf8cabcffe285ecc435ec2eeddd09fd4ae214ccc7c"} Nov 01 23:16:38.635988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.635962 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:38.698926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.698892 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f5a86e-8126-402f-90dc-e64752437f8e-kube-api-access-qbjgn\" (UniqueName: \"kubernetes.io/projected/b1f5a86e-8126-402f-90dc-e64752437f8e-kube-api-access-qbjgn\") pod \"b1f5a86e-8126-402f-90dc-e64752437f8e\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " Nov 01 23:16:38.699124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.698975 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-xks-ca\") pod \"b1f5a86e-8126-402f-90dc-e64752437f8e\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " Nov 01 23:16:38.699124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.699025 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-openshift-ca\") pod \"b1f5a86e-8126-402f-90dc-e64752437f8e\" (UID: \"b1f5a86e-8126-402f-90dc-e64752437f8e\") " Nov 01 23:16:38.699361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.699337 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-openshift-ca" pod "b1f5a86e-8126-402f-90dc-e64752437f8e" (UID: "b1f5a86e-8126-402f-90dc-e64752437f8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:16:38.701064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.701041 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f5a86e-8126-402f-90dc-e64752437f8e-kube-api-access-qbjgn" pod "b1f5a86e-8126-402f-90dc-e64752437f8e" (UID: "b1f5a86e-8126-402f-90dc-e64752437f8e"). InnerVolumeSpecName "kube-api-access-qbjgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:16:38.701160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.701064 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-xks-ca" pod "b1f5a86e-8126-402f-90dc-e64752437f8e" (UID: "b1f5a86e-8126-402f-90dc-e64752437f8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:16:38.799780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.799706 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjgn\" (UniqueName: \"kubernetes.io/projected/b1f5a86e-8126-402f-90dc-e64752437f8e-kube-api-access-qbjgn\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:38.799780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.799738 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:38.799780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:38.799771 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f5a86e-8126-402f-90dc-e64752437f8e-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:39.507942 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:39.507907 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" event={"ID":"b1f5a86e-8126-402f-90dc-e64752437f8e","Type":"ContainerDied","Data":"b51cbefee5f328b74f0bdbfdff9469b04de84f737967db07eb2180c635f1f72f"} Nov 01 23:16:39.507942 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:39.507936 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g" Nov 01 23:16:39.508163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:39.507956 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51cbefee5f328b74f0bdbfdff9469b04de84f737967db07eb2180c635f1f72f" Nov 01 23:16:44.523122 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:16:44.523081 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:16:46.318829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.318798 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799"] Nov 01 23:16:46.319311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.319242 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b1f5a86e-8126-402f-90dc-e64752437f8e" containerName="revoke-keys" Nov 01 23:16:46.319311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.319254 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f5a86e-8126-402f-90dc-e64752437f8e" containerName="revoke-keys" Nov 01 23:16:46.319439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.319323 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f5a86e-8126-402f-90dc-e64752437f8e" containerName="revoke-keys" Nov 01 23:16:46.322879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.322858 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.328352 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.328325 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:16:46.328545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.328392 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:16:46.328712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.328694 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:16:46.328840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.328773 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:16:46.332114 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.332092 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799"] Nov 01 23:16:46.364444 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.364413 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.364636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.364464 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.364636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.364605 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwt4n\" (UniqueName: \"kubernetes.io/projected/8fb1789e-eece-4e37-98a7-9c13993c266b-kube-api-access-zwt4n\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.465719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.465669 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.465905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.465747 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwt4n\" (UniqueName: \"kubernetes.io/projected/8fb1789e-eece-4e37-98a7-9c13993c266b-kube-api-access-zwt4n\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.465905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.465848 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.466387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.466364 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.467859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.467833 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.473551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.473531 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwt4n\" (UniqueName: \"kubernetes.io/projected/8fb1789e-eece-4e37-98a7-9c13993c266b-kube-api-access-zwt4n\") pod \"maas-api-revoke-sub-3d4156f1-hs799\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.638376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.638294 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:46.916957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:46.916930 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799"] Nov 01 23:16:46.919698 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:16:46.919653 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb1789e_eece_4e37_98a7_9c13993c266b.slice/crio-b3444f6b0d96f3fc8bf312ce6070f71b55c394a8925f17026a79962c529263ae WatchSource:0}: Error finding container b3444f6b0d96f3fc8bf312ce6070f71b55c394a8925f17026a79962c529263ae: Status 404 returned error can't find the container with id b3444f6b0d96f3fc8bf312ce6070f71b55c394a8925f17026a79962c529263ae Nov 01 23:16:47.567027 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:47.566934 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" event={"ID":"8fb1789e-eece-4e37-98a7-9c13993c266b","Type":"ContainerStarted","Data":"9a70768742648ace79799bba88e6bad99e3abcc3ff2f8c12c07eb612ed7e55e4"} Nov 01 23:16:47.567027 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:47.566989 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" event={"ID":"8fb1789e-eece-4e37-98a7-9c13993c266b","Type":"ContainerStarted","Data":"b3444f6b0d96f3fc8bf312ce6070f71b55c394a8925f17026a79962c529263ae"} Nov 01 23:16:47.585188 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:47.585137 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" podStartSLOduration=1.585123702 podStartE2EDuration="1.585123702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:16:47.582857351 +0000 UTC m=+3173.010735282" watchObservedRunningTime="2026-11-01 23:16:47.585123702 +0000 UTC m=+3173.013001632" Nov 01 23:16:48.574964 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:48.574926 4981 generic.go:356] "Generic (PLEG): container finished" podID="8fb1789e-eece-4e37-98a7-9c13993c266b" containerID="9a70768742648ace79799bba88e6bad99e3abcc3ff2f8c12c07eb612ed7e55e4" exitCode=0 Nov 01 23:16:48.575390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:48.575012 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" event={"ID":"8fb1789e-eece-4e37-98a7-9c13993c266b","Type":"ContainerDied","Data":"9a70768742648ace79799bba88e6bad99e3abcc3ff2f8c12c07eb612ed7e55e4"} Nov 01 23:16:49.721092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.721065 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:49.799444 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.799413 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-openshift-ca\") pod \"8fb1789e-eece-4e37-98a7-9c13993c266b\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " Nov 01 23:16:49.799444 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.799448 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-xks-ca\") pod \"8fb1789e-eece-4e37-98a7-9c13993c266b\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " Nov 01 23:16:49.799752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.799621 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb1789e-eece-4e37-98a7-9c13993c266b-kube-api-access-zwt4n\" (UniqueName: \"kubernetes.io/projected/8fb1789e-eece-4e37-98a7-9c13993c266b-kube-api-access-zwt4n\") pod \"8fb1789e-eece-4e37-98a7-9c13993c266b\" (UID: \"8fb1789e-eece-4e37-98a7-9c13993c266b\") " Nov 01 23:16:49.799876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.799851 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-openshift-ca" pod "8fb1789e-eece-4e37-98a7-9c13993c266b" (UID: "8fb1789e-eece-4e37-98a7-9c13993c266b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:16:49.801630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.801600 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb1789e-eece-4e37-98a7-9c13993c266b-kube-api-access-zwt4n" pod "8fb1789e-eece-4e37-98a7-9c13993c266b" (UID: "8fb1789e-eece-4e37-98a7-9c13993c266b"). InnerVolumeSpecName "kube-api-access-zwt4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:16:49.801630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.801619 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-xks-ca" pod "8fb1789e-eece-4e37-98a7-9c13993c266b" (UID: "8fb1789e-eece-4e37-98a7-9c13993c266b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:16:49.900482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.900407 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwt4n\" (UniqueName: \"kubernetes.io/projected/8fb1789e-eece-4e37-98a7-9c13993c266b-kube-api-access-zwt4n\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:49.900482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.900436 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:49.900482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:49.900452 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb1789e-eece-4e37-98a7-9c13993c266b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:16:50.021360 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.021327 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5"] Nov 01 23:16:50.026112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.026082 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b"] Nov 01 23:16:50.028115 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.028097 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149987a3-hkph5"] Nov 01 23:16:50.029778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.029759 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe424e47-qg46b"] Nov 01 23:16:50.588302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.588275 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" Nov 01 23:16:50.588302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.588298 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799" event={"ID":"8fb1789e-eece-4e37-98a7-9c13993c266b","Type":"ContainerDied","Data":"b3444f6b0d96f3fc8bf312ce6070f71b55c394a8925f17026a79962c529263ae"} Nov 01 23:16:50.588542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.588341 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3444f6b0d96f3fc8bf312ce6070f71b55c394a8925f17026a79962c529263ae" Nov 01 23:16:50.732413 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.732381 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a487583-0870-4da6-b967-5ec01928d764" path="/var/lib/kubelet/pods/9a487583-0870-4da6-b967-5ec01928d764/volumes" Nov 01 23:16:50.732840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:16:50.732810 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d3dcc1-b6ee-4998-9aba-1d76febe1de5" path="/var/lib/kubelet/pods/a0d3dcc1-b6ee-4998-9aba-1d76febe1de5/volumes" Nov 01 23:17:00.108023 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:00.107975 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:17:00.108747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:00.108698 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:17:02.281610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:02.280697 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:17:02.281610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:02.280785 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:17:03.846384 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.846352 4981 scope.go:122] "RemoveContainer" containerID="dedf1e800ee5d30ee80797fa21eb88bafc1b14905976102894577fa363ea3c30" Nov 01 23:17:03.855658 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.855639 4981 scope.go:122] "RemoveContainer" containerID="6bf327902a76a08630be3f6926568544a341837611679b44359240a91ccd06c0" Nov 01 23:17:03.863645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.863626 4981 scope.go:122] "RemoveContainer" containerID="e5095be26cf04e62a4d0873d9a191590fd214f043911b0d582eeed17ee7d38e0" Nov 01 23:17:03.871677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.871648 4981 scope.go:122] "RemoveContainer" containerID="97eb21d7537757009ec44276de98c882b38c06f773497a03c2085f40dea952ee" Nov 01 23:17:03.880326 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.880310 4981 scope.go:122] "RemoveContainer" containerID="a53cb9bbe1ddfc847006250e580b0998817ad49816a216f57aa22fcd7ce199f2" Nov 01 23:17:03.888438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.888416 4981 scope.go:122] "RemoveContainer" containerID="191947c6d92a2b468a0013dc121e63b40b55ab6b4ed1cee5f7fa97ae5794be39" Nov 01 23:17:03.896225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.896207 4981 scope.go:122] "RemoveContainer" containerID="fadc87331c5b7b30c1193f9d22494b6e085ca06fff45b1b6bc12b776b2ea9d3c" Nov 01 23:17:03.903840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.903825 4981 scope.go:122] "RemoveContainer" containerID="d064c1c3f05b784e7d428179e37ff2a4086e2b8677880a0fc68f611ad61f2978" Nov 01 23:17:03.911448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.911427 4981 scope.go:122] "RemoveContainer" containerID="f836dc55129f503f13f79fd2cadb147cbc11ba2a485154179d7bb711a85135da" Nov 01 23:17:03.919949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.919876 4981 scope.go:122] "RemoveContainer" containerID="be4b7645b52fa6c7265add943e594fb5af846b022814edaf687b961de69488c4" Nov 01 23:17:03.930121 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:03.930101 4981 scope.go:122] "RemoveContainer" containerID="899bdd9fa93b995912c922333eedd9b75fc34a26d848165fef7a4156342db50b" Nov 01 23:17:04.234083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.234046 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799"] Nov 01 23:17:04.240758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.240732 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hs799"] Nov 01 23:17:04.253961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.253923 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs"] Nov 01 23:17:04.254415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.254396 4981 cpu_manager.go:414] "containerMap: removing container" podUID="8fb1789e-eece-4e37-98a7-9c13993c266b" containerName="revoke-keys" Nov 01 23:17:04.254528 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.254445 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb1789e-eece-4e37-98a7-9c13993c266b" containerName="revoke-keys" Nov 01 23:17:04.254866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.254658 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb1789e-eece-4e37-98a7-9c13993c266b" containerName="revoke-keys" Nov 01 23:17:04.260074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.260056 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.264719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.264441 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:17:04.265458 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.265436 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:17:04.265615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.265479 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:17:04.265615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.265521 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:17:04.268635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.268590 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs"] Nov 01 23:17:04.393514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.393474 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.393514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.393514 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2hh\" (UniqueName: \"kubernetes.io/projected/4d635168-5f19-4f07-8519-038c9e51a6f9-kube-api-access-lq2hh\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.393756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.393668 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.494951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.494869 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.494951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.494905 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2hh\" (UniqueName: \"kubernetes.io/projected/4d635168-5f19-4f07-8519-038c9e51a6f9-kube-api-access-lq2hh\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.495160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.494957 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.495550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.495528 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.497287 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.497266 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.503153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.503131 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2hh\" (UniqueName: \"kubernetes.io/projected/4d635168-5f19-4f07-8519-038c9e51a6f9-kube-api-access-lq2hh\") pod \"maas-api-revoke-sub-3d4156f1-9s8hs\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.575215 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.575192 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:04.731884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.731854 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb1789e-eece-4e37-98a7-9c13993c266b" path="/var/lib/kubelet/pods/8fb1789e-eece-4e37-98a7-9c13993c266b/volumes" Nov 01 23:17:04.849222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:04.849199 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs"] Nov 01 23:17:04.850996 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:17:04.850968 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d635168_5f19_4f07_8519_038c9e51a6f9.slice/crio-c0e3585883313c7addacaf3d0fa25d204251207caded8e342976aa00c644b3c7 WatchSource:0}: Error finding container c0e3585883313c7addacaf3d0fa25d204251207caded8e342976aa00c644b3c7: Status 404 returned error can't find the container with id c0e3585883313c7addacaf3d0fa25d204251207caded8e342976aa00c644b3c7 Nov 01 23:17:05.698902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:05.698873 4981 generic.go:356] "Generic (PLEG): container finished" podID="4d635168-5f19-4f07-8519-038c9e51a6f9" containerID="391c2fa24a86587f0c171f6f510d02617a11cec9a74756d3d1e764227ec98abe" exitCode=0 Nov 01 23:17:05.699012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:05.698958 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" event={"ID":"4d635168-5f19-4f07-8519-038c9e51a6f9","Type":"ContainerDied","Data":"391c2fa24a86587f0c171f6f510d02617a11cec9a74756d3d1e764227ec98abe"} Nov 01 23:17:05.699075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:05.699012 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" event={"ID":"4d635168-5f19-4f07-8519-038c9e51a6f9","Type":"ContainerStarted","Data":"c0e3585883313c7addacaf3d0fa25d204251207caded8e342976aa00c644b3c7"} Nov 01 23:17:06.846506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:06.846480 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:06.918990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:06.918961 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-xks-ca\") pod \"4d635168-5f19-4f07-8519-038c9e51a6f9\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " Nov 01 23:17:06.919162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:06.919041 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-openshift-ca\") pod \"4d635168-5f19-4f07-8519-038c9e51a6f9\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " Nov 01 23:17:06.919162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:06.919151 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d635168-5f19-4f07-8519-038c9e51a6f9-kube-api-access-lq2hh\" (UniqueName: \"kubernetes.io/projected/4d635168-5f19-4f07-8519-038c9e51a6f9-kube-api-access-lq2hh\") pod \"4d635168-5f19-4f07-8519-038c9e51a6f9\" (UID: \"4d635168-5f19-4f07-8519-038c9e51a6f9\") " Nov 01 23:17:06.919398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:06.919375 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-openshift-ca" pod "4d635168-5f19-4f07-8519-038c9e51a6f9" (UID: "4d635168-5f19-4f07-8519-038c9e51a6f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:17:06.921052 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:06.921023 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-xks-ca" pod "4d635168-5f19-4f07-8519-038c9e51a6f9" (UID: "4d635168-5f19-4f07-8519-038c9e51a6f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:17:06.921160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:06.921105 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d635168-5f19-4f07-8519-038c9e51a6f9-kube-api-access-lq2hh" pod "4d635168-5f19-4f07-8519-038c9e51a6f9" (UID: "4d635168-5f19-4f07-8519-038c9e51a6f9"). InnerVolumeSpecName "kube-api-access-lq2hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:17:07.020681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:07.020608 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2hh\" (UniqueName: \"kubernetes.io/projected/4d635168-5f19-4f07-8519-038c9e51a6f9-kube-api-access-lq2hh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:07.020681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:07.020636 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:07.020681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:07.020645 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d635168-5f19-4f07-8519-038c9e51a6f9-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:07.711730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:07.711703 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" Nov 01 23:17:07.711912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:07.711702 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs" event={"ID":"4d635168-5f19-4f07-8519-038c9e51a6f9","Type":"ContainerDied","Data":"c0e3585883313c7addacaf3d0fa25d204251207caded8e342976aa00c644b3c7"} Nov 01 23:17:07.711912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:07.711827 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0e3585883313c7addacaf3d0fa25d204251207caded8e342976aa00c644b3c7" Nov 01 23:17:13.028937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:13.028900 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876"] Nov 01 23:17:13.035726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:13.035698 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d32c89d-bw876"] Nov 01 23:17:14.523555 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:17:14.523517 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:17:14.733515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:14.733483 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0911c3c-3b05-4f9b-8de7-5f21dd85c707" path="/var/lib/kubelet/pods/c0911c3c-3b05-4f9b-8de7-5f21dd85c707/volumes" Nov 01 23:17:15.025564 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:15.025518 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw"] Nov 01 23:17:15.028121 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:15.028084 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg"] Nov 01 23:17:15.032108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:15.032082 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7293b400-q75xw"] Nov 01 23:17:15.033781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:15.033757 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded41bf4-4x5kg"] Nov 01 23:17:15.787057 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:15.787026 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:17:16.150222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:16.150146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:17:16.732439 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:16.732407 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d1d9b2-08ca-4827-a6ff-b35966fc95ba" path="/var/lib/kubelet/pods/38d1d9b2-08ca-4827-a6ff-b35966fc95ba/volumes" Nov 01 23:17:16.732825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:16.732812 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741ceb40-f226-49d0-b06a-2d1acf59a08f" path="/var/lib/kubelet/pods/741ceb40-f226-49d0-b06a-2d1acf59a08f/volumes" Nov 01 23:17:32.501415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.501376 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt"] Nov 01 23:17:32.501895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.501853 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4d635168-5f19-4f07-8519-038c9e51a6f9" containerName="revoke-keys" Nov 01 23:17:32.501895 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.501866 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d635168-5f19-4f07-8519-038c9e51a6f9" containerName="revoke-keys" Nov 01 23:17:32.501976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.501936 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d635168-5f19-4f07-8519-038c9e51a6f9" containerName="revoke-keys" Nov 01 23:17:32.505077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.505059 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.511124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.511081 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:17:32.511261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.511136 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:17:32.511261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.511226 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:17:32.511368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.511354 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:17:32.516602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.514194 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt"] Nov 01 23:17:32.592008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.591980 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.592174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.592122 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.592221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.592174 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xtdw\" (UniqueName: \"kubernetes.io/projected/c575fa69-472e-41bf-b046-6d3b913025a1-kube-api-access-9xtdw\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.692980 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.692946 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.693142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.692998 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xtdw\" (UniqueName: \"kubernetes.io/projected/c575fa69-472e-41bf-b046-6d3b913025a1-kube-api-access-9xtdw\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.693142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.693032 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.693719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.693691 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.695259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.695229 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.701315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.701288 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xtdw\" (UniqueName: \"kubernetes.io/projected/c575fa69-472e-41bf-b046-6d3b913025a1-kube-api-access-9xtdw\") pod \"maas-api-revoke-sub-e2fc9c5b-rs9qt\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:32.820858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:32.820794 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:33.092108 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:33.092080 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt"] Nov 01 23:17:33.093918 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:17:33.093884 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc575fa69_472e_41bf_b046_6d3b913025a1.slice/crio-059a784201098bdd3b13d1807f0598eb71984b62956234b5e56c57655b2346c8 WatchSource:0}: Error finding container 059a784201098bdd3b13d1807f0598eb71984b62956234b5e56c57655b2346c8: Status 404 returned error can't find the container with id 059a784201098bdd3b13d1807f0598eb71984b62956234b5e56c57655b2346c8 Nov 01 23:17:33.877620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:33.877565 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" event={"ID":"c575fa69-472e-41bf-b046-6d3b913025a1","Type":"ContainerStarted","Data":"56f8ad47d0ee2b2ad46339f13f18fefc88ba727d6e67d749b71406aa3ecd70f5"} Nov 01 23:17:33.877994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:33.877655 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" event={"ID":"c575fa69-472e-41bf-b046-6d3b913025a1","Type":"ContainerStarted","Data":"059a784201098bdd3b13d1807f0598eb71984b62956234b5e56c57655b2346c8"} Nov 01 23:17:33.897209 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:33.897167 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" podStartSLOduration=1.897153605 podStartE2EDuration="1.897153605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:33.895668079 +0000 UTC m=+3219.323546010" watchObservedRunningTime="2026-11-01 23:17:33.897153605 +0000 UTC m=+3219.325031536" Nov 01 23:17:34.884806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:34.884763 4981 generic.go:356] "Generic (PLEG): container finished" podID="c575fa69-472e-41bf-b046-6d3b913025a1" containerID="56f8ad47d0ee2b2ad46339f13f18fefc88ba727d6e67d749b71406aa3ecd70f5" exitCode=0 Nov 01 23:17:34.885264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:34.884856 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" event={"ID":"c575fa69-472e-41bf-b046-6d3b913025a1","Type":"ContainerDied","Data":"56f8ad47d0ee2b2ad46339f13f18fefc88ba727d6e67d749b71406aa3ecd70f5"} Nov 01 23:17:36.031632 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.031606 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:36.122622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.122563 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-openshift-ca\") pod \"c575fa69-472e-41bf-b046-6d3b913025a1\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " Nov 01 23:17:36.122803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.122681 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c575fa69-472e-41bf-b046-6d3b913025a1-kube-api-access-9xtdw\" (UniqueName: \"kubernetes.io/projected/c575fa69-472e-41bf-b046-6d3b913025a1-kube-api-access-9xtdw\") pod \"c575fa69-472e-41bf-b046-6d3b913025a1\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " Nov 01 23:17:36.122803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.122717 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-xks-ca\") pod \"c575fa69-472e-41bf-b046-6d3b913025a1\" (UID: \"c575fa69-472e-41bf-b046-6d3b913025a1\") " Nov 01 23:17:36.123000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.122978 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-openshift-ca" pod "c575fa69-472e-41bf-b046-6d3b913025a1" (UID: "c575fa69-472e-41bf-b046-6d3b913025a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:17:36.123120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.123096 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:36.124734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.124711 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-xks-ca" pod "c575fa69-472e-41bf-b046-6d3b913025a1" (UID: "c575fa69-472e-41bf-b046-6d3b913025a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:17:36.124870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.124856 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c575fa69-472e-41bf-b046-6d3b913025a1-kube-api-access-9xtdw" pod "c575fa69-472e-41bf-b046-6d3b913025a1" (UID: "c575fa69-472e-41bf-b046-6d3b913025a1"). InnerVolumeSpecName "kube-api-access-9xtdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:17:36.224533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.224497 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xtdw\" (UniqueName: \"kubernetes.io/projected/c575fa69-472e-41bf-b046-6d3b913025a1-kube-api-access-9xtdw\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:36.224533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.224527 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c575fa69-472e-41bf-b046-6d3b913025a1-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:36.896629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.896589 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" event={"ID":"c575fa69-472e-41bf-b046-6d3b913025a1","Type":"ContainerDied","Data":"059a784201098bdd3b13d1807f0598eb71984b62956234b5e56c57655b2346c8"} Nov 01 23:17:36.896818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.896653 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="059a784201098bdd3b13d1807f0598eb71984b62956234b5e56c57655b2346c8" Nov 01 23:17:36.896818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:36.896603 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt" Nov 01 23:17:37.396003 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:37.395953 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:17:41.028369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:41.028326 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv"] Nov 01 23:17:41.034764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:41.034735 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f5539c1-4bkgv"] Nov 01 23:17:42.689315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.689272 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h"] Nov 01 23:17:42.689932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.689911 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c575fa69-472e-41bf-b046-6d3b913025a1" containerName="revoke-keys" Nov 01 23:17:42.690024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.689934 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c575fa69-472e-41bf-b046-6d3b913025a1" containerName="revoke-keys" Nov 01 23:17:42.690096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.690074 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c575fa69-472e-41bf-b046-6d3b913025a1" containerName="revoke-keys" Nov 01 23:17:42.693158 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.693137 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.697913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.697891 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:17:42.698952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.698933 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:17:42.700607 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.700566 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h"] Nov 01 23:17:42.734437 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.734407 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebde4c11-a30e-48af-92bf-943ef2e40fdb" path="/var/lib/kubelet/pods/ebde4c11-a30e-48af-92bf-943ef2e40fdb/volumes" Nov 01 23:17:42.765429 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.765400 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:17:42.765429 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.765405 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:17:42.786919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.786889 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.787033 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.786958 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hrh\" (UniqueName: \"kubernetes.io/projected/e559ead6-5242-4170-9871-7a45987b3eb3-kube-api-access-82hrh\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.787092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.787046 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.888126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.888093 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hrh\" (UniqueName: \"kubernetes.io/projected/e559ead6-5242-4170-9871-7a45987b3eb3-kube-api-access-82hrh\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.888319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.888144 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.888319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.888199 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.888828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.888808 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.890566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.890547 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:42.896480 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:42.896459 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hrh\" (UniqueName: \"kubernetes.io/projected/e559ead6-5242-4170-9871-7a45987b3eb3-kube-api-access-82hrh\") pod \"maas-api-revoke-sub-5a29efb3-jtj2h\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:43.009001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:43.008914 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:43.282566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:43.282539 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h"] Nov 01 23:17:43.284319 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:17:43.284287 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode559ead6_5242_4170_9871_7a45987b3eb3.slice/crio-3d6a940a9aea93f6bde1f54266661a63d2a3e8a0cee326faebf791928668e466 WatchSource:0}: Error finding container 3d6a940a9aea93f6bde1f54266661a63d2a3e8a0cee326faebf791928668e466: Status 404 returned error can't find the container with id 3d6a940a9aea93f6bde1f54266661a63d2a3e8a0cee326faebf791928668e466 Nov 01 23:17:43.942135 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:43.942097 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" event={"ID":"e559ead6-5242-4170-9871-7a45987b3eb3","Type":"ContainerStarted","Data":"274cad31625790620621a30b410f548d67380cad2dbed9cceacb155ee717c266"} Nov 01 23:17:43.942705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:43.942152 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" event={"ID":"e559ead6-5242-4170-9871-7a45987b3eb3","Type":"ContainerStarted","Data":"3d6a940a9aea93f6bde1f54266661a63d2a3e8a0cee326faebf791928668e466"} Nov 01 23:17:43.957981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:43.957927 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" podStartSLOduration=1.9579100820000002 podStartE2EDuration="1.957910082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:43.956890264 +0000 UTC m=+3229.384768211" watchObservedRunningTime="2026-11-01 23:17:43.957910082 +0000 UTC m=+3229.385788015" Nov 01 23:17:44.530770 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:17:44.530726 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:17:44.955356 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:44.955311 4981 generic.go:356] "Generic (PLEG): container finished" podID="e559ead6-5242-4170-9871-7a45987b3eb3" containerID="274cad31625790620621a30b410f548d67380cad2dbed9cceacb155ee717c266" exitCode=0 Nov 01 23:17:44.955922 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:44.955503 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" event={"ID":"e559ead6-5242-4170-9871-7a45987b3eb3","Type":"ContainerDied","Data":"274cad31625790620621a30b410f548d67380cad2dbed9cceacb155ee717c266"} Nov 01 23:17:46.306084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.306057 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:46.356533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.356488 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-xks-ca\") pod \"e559ead6-5242-4170-9871-7a45987b3eb3\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " Nov 01 23:17:46.356756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.356654 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e559ead6-5242-4170-9871-7a45987b3eb3-kube-api-access-82hrh\" (UniqueName: \"kubernetes.io/projected/e559ead6-5242-4170-9871-7a45987b3eb3-kube-api-access-82hrh\") pod \"e559ead6-5242-4170-9871-7a45987b3eb3\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " Nov 01 23:17:46.359771 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.359686 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-openshift-ca\") pod \"e559ead6-5242-4170-9871-7a45987b3eb3\" (UID: \"e559ead6-5242-4170-9871-7a45987b3eb3\") " Nov 01 23:17:46.360141 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.360112 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-openshift-ca" pod "e559ead6-5242-4170-9871-7a45987b3eb3" (UID: "e559ead6-5242-4170-9871-7a45987b3eb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:17:46.360768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.360741 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:46.364878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.364845 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-xks-ca" pod "e559ead6-5242-4170-9871-7a45987b3eb3" (UID: "e559ead6-5242-4170-9871-7a45987b3eb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:17:46.366246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.366213 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e559ead6-5242-4170-9871-7a45987b3eb3-kube-api-access-82hrh" pod "e559ead6-5242-4170-9871-7a45987b3eb3" (UID: "e559ead6-5242-4170-9871-7a45987b3eb3"). InnerVolumeSpecName "kube-api-access-82hrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:17:46.463876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.463842 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e559ead6-5242-4170-9871-7a45987b3eb3-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:46.463876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.463871 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hrh\" (UniqueName: \"kubernetes.io/projected/e559ead6-5242-4170-9871-7a45987b3eb3-kube-api-access-82hrh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:17:46.967981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.967953 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" Nov 01 23:17:46.968138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.967954 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h" event={"ID":"e559ead6-5242-4170-9871-7a45987b3eb3","Type":"ContainerDied","Data":"3d6a940a9aea93f6bde1f54266661a63d2a3e8a0cee326faebf791928668e466"} Nov 01 23:17:46.968138 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:46.968101 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d6a940a9aea93f6bde1f54266661a63d2a3e8a0cee326faebf791928668e466" Nov 01 23:17:57.811472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.811438 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs"] Nov 01 23:17:57.816732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.816705 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9s8hs"] Nov 01 23:17:57.832470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.832442 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb"] Nov 01 23:17:57.832890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.832876 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e559ead6-5242-4170-9871-7a45987b3eb3" containerName="revoke-keys" Nov 01 23:17:57.832941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.832893 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e559ead6-5242-4170-9871-7a45987b3eb3" containerName="revoke-keys" Nov 01 23:17:57.832981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.832963 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e559ead6-5242-4170-9871-7a45987b3eb3" containerName="revoke-keys" Nov 01 23:17:57.835992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.835977 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.840126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.840104 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:17:57.842693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.841786 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:17:57.842693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.841992 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:17:57.852980 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.846785 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb"] Nov 01 23:17:57.861378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.843059 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:17:57.872493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.872456 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtjc\" (UniqueName: \"kubernetes.io/projected/a6876a52-ba28-469f-8a0b-863571c5db71-kube-api-access-wbtjc\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.872649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.872623 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.872732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.872675 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.974303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.974262 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.974303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.974306 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.974542 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.974381 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtjc\" (UniqueName: \"kubernetes.io/projected/a6876a52-ba28-469f-8a0b-863571c5db71-kube-api-access-wbtjc\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.975077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.975051 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.976721 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.976698 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:57.983278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:57.983255 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtjc\" (UniqueName: \"kubernetes.io/projected/a6876a52-ba28-469f-8a0b-863571c5db71-kube-api-access-wbtjc\") pod \"maas-api-revoke-sub-3d4156f1-vklvb\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:58.160265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:58.160189 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:17:58.436951 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:58.436910 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb"] Nov 01 23:17:58.440751 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:17:58.440725 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6876a52_ba28_469f_8a0b_863571c5db71.slice/crio-68a3dab8076adb242dc54d0de52e748f88fc897ede25c29fbeb84df8b03fb555 WatchSource:0}: Error finding container 68a3dab8076adb242dc54d0de52e748f88fc897ede25c29fbeb84df8b03fb555: Status 404 returned error can't find the container with id 68a3dab8076adb242dc54d0de52e748f88fc897ede25c29fbeb84df8b03fb555 Nov 01 23:17:58.732303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:58.732230 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d635168-5f19-4f07-8519-038c9e51a6f9" path="/var/lib/kubelet/pods/4d635168-5f19-4f07-8519-038c9e51a6f9/volumes" Nov 01 23:17:59.037175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:59.037082 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" event={"ID":"a6876a52-ba28-469f-8a0b-863571c5db71","Type":"ContainerStarted","Data":"f2a4baae13da0cf182c1c786157e4b49fb9aac42d5a87b65cbc8b53dd108e410"} Nov 01 23:17:59.037175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:59.037155 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" event={"ID":"a6876a52-ba28-469f-8a0b-863571c5db71","Type":"ContainerStarted","Data":"68a3dab8076adb242dc54d0de52e748f88fc897ede25c29fbeb84df8b03fb555"} Nov 01 23:17:59.053485 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:17:59.053437 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" podStartSLOduration=2.053424164 podStartE2EDuration="2.053424164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:59.052306488 +0000 UTC m=+3244.480184420" watchObservedRunningTime="2026-11-01 23:17:59.053424164 +0000 UTC m=+3244.481302094" Nov 01 23:18:00.044370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:00.044337 4981 generic.go:356] "Generic (PLEG): container finished" podID="a6876a52-ba28-469f-8a0b-863571c5db71" containerID="f2a4baae13da0cf182c1c786157e4b49fb9aac42d5a87b65cbc8b53dd108e410" exitCode=0 Nov 01 23:18:00.044904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:00.044424 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" event={"ID":"a6876a52-ba28-469f-8a0b-863571c5db71","Type":"ContainerDied","Data":"f2a4baae13da0cf182c1c786157e4b49fb9aac42d5a87b65cbc8b53dd108e410"} Nov 01 23:18:01.187736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.187710 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:18:01.309540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.309450 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6876a52-ba28-469f-8a0b-863571c5db71-kube-api-access-wbtjc\" (UniqueName: \"kubernetes.io/projected/a6876a52-ba28-469f-8a0b-863571c5db71-kube-api-access-wbtjc\") pod \"a6876a52-ba28-469f-8a0b-863571c5db71\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " Nov 01 23:18:01.309768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.309544 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-xks-ca\") pod \"a6876a52-ba28-469f-8a0b-863571c5db71\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " Nov 01 23:18:01.309768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.309653 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-openshift-ca\") pod \"a6876a52-ba28-469f-8a0b-863571c5db71\" (UID: \"a6876a52-ba28-469f-8a0b-863571c5db71\") " Nov 01 23:18:01.310088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.310064 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-openshift-ca" pod "a6876a52-ba28-469f-8a0b-863571c5db71" (UID: "a6876a52-ba28-469f-8a0b-863571c5db71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:18:01.311690 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.311666 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-xks-ca" pod "a6876a52-ba28-469f-8a0b-863571c5db71" (UID: "a6876a52-ba28-469f-8a0b-863571c5db71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:18:01.311764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.311738 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6876a52-ba28-469f-8a0b-863571c5db71-kube-api-access-wbtjc" pod "a6876a52-ba28-469f-8a0b-863571c5db71" (UID: "a6876a52-ba28-469f-8a0b-863571c5db71"). InnerVolumeSpecName "kube-api-access-wbtjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:18:01.410884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.410837 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbtjc\" (UniqueName: \"kubernetes.io/projected/a6876a52-ba28-469f-8a0b-863571c5db71-kube-api-access-wbtjc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:18:01.410884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.410875 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:18:01.410884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:01.410891 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6876a52-ba28-469f-8a0b-863571c5db71-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:18:02.057263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:02.057228 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" Nov 01 23:18:02.057263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:02.057254 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb" event={"ID":"a6876a52-ba28-469f-8a0b-863571c5db71","Type":"ContainerDied","Data":"68a3dab8076adb242dc54d0de52e748f88fc897ede25c29fbeb84df8b03fb555"} Nov 01 23:18:02.057505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:02.057302 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68a3dab8076adb242dc54d0de52e748f88fc897ede25c29fbeb84df8b03fb555" Nov 01 23:18:03.991202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:03.991166 4981 scope.go:122] "RemoveContainer" containerID="6fa44991b973164900d4a2dbe4dddba34466ad761e27975a51b5751a769c8e6a" Nov 01 23:18:04.000178 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.000145 4981 scope.go:122] "RemoveContainer" containerID="1975ca0aa780c22af9932c11c03d12d2dc9b6dd5a68dfc37e3ba8219f7ca2eb8" Nov 01 23:18:04.008781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.008761 4981 scope.go:122] "RemoveContainer" containerID="f88fcbe9249d34a82e4b458b3a4c79da1eae9a465f85d55c3493195dea155b72" Nov 01 23:18:04.017337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.017308 4981 scope.go:122] "RemoveContainer" containerID="29a719cc8ab90633bdb35674da03095752b498c436d3d287a1e467199e3ddd21" Nov 01 23:18:04.025928 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.025902 4981 scope.go:122] "RemoveContainer" containerID="618f86c9c82999794bdd7ffc7d8c43f258fe86c687172e4ed6665c043b7a929b" Nov 01 23:18:04.034668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.034645 4981 scope.go:122] "RemoveContainer" containerID="24e28925ecfe5fbb340b749a21ded7a33bfa38ef13ef2abc895add8c30f03476" Nov 01 23:18:04.043065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.043043 4981 scope.go:122] "RemoveContainer" containerID="d761416ba996277ba782eaf0d3ec6605001d0c7f8f8b294cd2d68602c88f0312" Nov 01 23:18:04.051428 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.051401 4981 scope.go:122] "RemoveContainer" containerID="6dc5ebefbd68e08c0e089afd8632cad888423cda63f5746152fa14a567aeaf53" Nov 01 23:18:04.059752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.059728 4981 scope.go:122] "RemoveContainer" containerID="4c31314af1692dc7f23ba44f8731e237e3502089c5d1ec2a5c112037af86b4dc" Nov 01 23:18:04.068555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.068535 4981 scope.go:122] "RemoveContainer" containerID="45e8747e9a43077d9d8e826cfd27928bac47d128934dc60751a6bb50469bf34f" Nov 01 23:18:04.077698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.077679 4981 scope.go:122] "RemoveContainer" containerID="18b6b9c2637b5300a26a94d61017b690c2c9140e5caf19a7ee9f6435c2737768" Nov 01 23:18:04.085892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.085871 4981 scope.go:122] "RemoveContainer" containerID="c0b3357551d61159a19984a26bb808a6cc35ffcaf3827f15fd1a01f866a76674" Nov 01 23:18:04.094498 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:04.094474 4981 scope.go:122] "RemoveContainer" containerID="fb2879ca89ce60cabce5b317bfe64bbf9ef861a6e7252078d49bd058eb6cc4a1" Nov 01 23:18:11.029426 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:11.029229 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:14.523439 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:18:14.523394 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:18:15.826927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:15.826893 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:15.927514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:15.927485 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:16.237729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.237700 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:16.313514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.313477 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj"] Nov 01 23:18:16.313958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.313944 4981 cpu_manager.go:414] "containerMap: removing container" podUID="a6876a52-ba28-469f-8a0b-863571c5db71" containerName="revoke-keys" Nov 01 23:18:16.314023 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.313960 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6876a52-ba28-469f-8a0b-863571c5db71" containerName="revoke-keys" Nov 01 23:18:16.314092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.314036 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6876a52-ba28-469f-8a0b-863571c5db71" containerName="revoke-keys" Nov 01 23:18:16.320543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.320519 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.329465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.329175 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj"] Nov 01 23:18:16.332160 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.332133 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-aaf3e1-openshift-default-dockercfg-x5k64\"" Nov 01 23:18:16.371065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.371033 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:16.392079 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392043 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5aa6d74-fa20-4203-935a-60be18072fa3-istiod-ca-cert\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392087 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-token\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392122 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-data\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392182 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-credential-socket\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392291 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-podinfo\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392377 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392328 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-envoy\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392418 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-socket\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392498 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392466 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n58tg\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-kube-api-access-n58tg\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.392540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.392501 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-certs\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.492996 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-certs\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493045 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5aa6d74-fa20-4203-935a-60be18072fa3-istiod-ca-cert\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493070 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-token\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493092 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-data\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493117 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-credential-socket\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493193 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-podinfo\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493233 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-envoy\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493483 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-socket\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493490 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-data\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493629 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493548 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n58tg\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-kube-api-access-n58tg\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493705 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-credential-socket\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493791 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-socket\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.493848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.493826 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-certs\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.494216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.494186 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5aa6d74-fa20-4203-935a-60be18072fa3-istiod-ca-cert\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.495608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.495559 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-envoy\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.496034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.496010 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-podinfo\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.501306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.501275 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-token\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.502055 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.501990 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n58tg\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-kube-api-access-n58tg\") pod \"e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.503874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.503351 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:16.636605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.636544 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:16.911410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.911385 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj"] Nov 01 23:18:16.913481 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:18:16.913455 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aa6d74_fa20_4203_935a_60be18072fa3.slice/crio-cb71ed33b25d2c74460f52a08e49b631591e2bf631c3346e415497ef89389db3 WatchSource:0}: Error finding container cb71ed33b25d2c74460f52a08e49b631591e2bf631c3346e415497ef89389db3: Status 404 returned error can't find the container with id cb71ed33b25d2c74460f52a08e49b631591e2bf631c3346e415497ef89389db3 Nov 01 23:18:16.915667 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.915635 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:18:16.915787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.915713 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:18:16.915787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:16.915763 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:18:17.179124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:17.179085 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" event={"ID":"c5aa6d74-fa20-4203-935a-60be18072fa3","Type":"ContainerStarted","Data":"72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136"} Nov 01 23:18:17.179310 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:17.179146 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" event={"ID":"c5aa6d74-fa20-4203-935a-60be18072fa3","Type":"ContainerStarted","Data":"cb71ed33b25d2c74460f52a08e49b631591e2bf631c3346e415497ef89389db3"} Nov 01 23:18:17.202167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:17.202116 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podStartSLOduration=1.202102464 podStartE2EDuration="1.202102464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:17.200229505 +0000 UTC m=+3262.628107437" watchObservedRunningTime="2026-11-01 23:18:17.202102464 +0000 UTC m=+3262.629980395" Nov 01 23:18:17.636800 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:17.636707 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:17.642203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:17.642177 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:18.184835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:18.184799 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:18.185816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:18.185794 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:19.316526 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:19.315985 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:19.418871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:19.418839 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:20.473514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.473480 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm"] Nov 01 23:18:20.477509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.477487 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.480240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.480212 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:20.487806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.487781 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:18:20.487993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.487832 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-aaf3e1-serving-cert\"" Nov 01 23:18:20.488130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.487978 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:18:20.488204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.488077 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:18:20.488204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.488153 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 23:18:20.490361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.490336 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm"] Nov 01 23:18:20.521787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.521751 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87kt9\" (UniqueName: \"kubernetes.io/projected/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-kube-api-access-87kt9\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.522006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.521808 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.522006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.521958 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.522151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.522022 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.575267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.575226 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:20.623361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.623321 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87kt9\" (UniqueName: \"kubernetes.io/projected/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-kube-api-access-87kt9\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.624197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.624100 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.624197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.624183 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.624515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.624241 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.626000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.625972 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.626734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.626706 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk"] Nov 01 23:18:20.627912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.627883 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.628773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.628752 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-metrics-certs\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.630806 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.630782 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:20.633122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.633098 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87kt9\" (UniqueName: \"kubernetes.io/projected/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-kube-api-access-87kt9\") pod \"maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.641296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.641269 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk"] Nov 01 23:18:20.703396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.703356 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-aaf3e1\"" Nov 01 23:18:20.703738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.703681 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-94d6703a-dockercfg-9pgtk\"" Nov 01 23:18:20.725147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.725060 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwhd\" (UniqueName: \"kubernetes.io/projected/ff54d54a-efce-4311-81e7-ccaf573d7487-kube-api-access-jnwhd\") pod \"payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:20.725319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.725219 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff54d54a-efce-4311-81e7-ccaf573d7487-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:20.796963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.796922 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:20.825750 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.825716 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff54d54a-efce-4311-81e7-ccaf573d7487-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:20.825959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.825760 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwhd\" (UniqueName: \"kubernetes.io/projected/ff54d54a-efce-4311-81e7-ccaf573d7487-kube-api-access-jnwhd\") pod \"payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:20.826518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.826488 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff54d54a-efce-4311-81e7-ccaf573d7487-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:20.834565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.834523 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwhd\" (UniqueName: \"kubernetes.io/projected/ff54d54a-efce-4311-81e7-ccaf573d7487-kube-api-access-jnwhd\") pod \"payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:20.964808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:20.964767 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:21.075567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:21.075542 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm"] Nov 01 23:18:21.076502 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:18:21.076480 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebc7c70_d6d9_49b5_9bfa_f66addaaa0b8.slice/crio-95c71a130abb76cb8877830f3863edea871035c70afc3d22ea74245502ac7d81 WatchSource:0}: Error finding container 95c71a130abb76cb8877830f3863edea871035c70afc3d22ea74245502ac7d81: Status 404 returned error can't find the container with id 95c71a130abb76cb8877830f3863edea871035c70afc3d22ea74245502ac7d81 Nov 01 23:18:21.077974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:21.077956 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:18:21.180876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:21.180842 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:21.201850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:21.201812 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" event={"ID":"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8","Type":"ContainerStarted","Data":"95c71a130abb76cb8877830f3863edea871035c70afc3d22ea74245502ac7d81"} Nov 01 23:18:21.243788 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:18:21.243749 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff54d54a_efce_4311_81e7_ccaf573d7487.slice/crio-e4f3e9dddfc48bb04b94267696925dcb709eba6ba21d31472207ede95240423d WatchSource:0}: Error finding container e4f3e9dddfc48bb04b94267696925dcb709eba6ba21d31472207ede95240423d: Status 404 returned error can't find the container with id e4f3e9dddfc48bb04b94267696925dcb709eba6ba21d31472207ede95240423d Nov 01 23:18:21.244163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:21.244137 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk"] Nov 01 23:18:21.865342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:21.865307 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:22.208444 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.208406 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" event={"ID":"ff54d54a-efce-4311-81e7-ccaf573d7487","Type":"ContainerStarted","Data":"c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b"} Nov 01 23:18:22.208688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.208473 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" event={"ID":"ff54d54a-efce-4311-81e7-ccaf573d7487","Type":"ContainerStarted","Data":"e4f3e9dddfc48bb04b94267696925dcb709eba6ba21d31472207ede95240423d"} Nov 01 23:18:22.208688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.208502 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:22.210091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.210067 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" event={"ID":"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8","Type":"ContainerStarted","Data":"9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd"} Nov 01 23:18:22.210200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.210111 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:22.227880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.227829 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" podStartSLOduration=2.227815697 podStartE2EDuration="2.227815697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:22.22541336 +0000 UTC m=+3267.653291288" watchObservedRunningTime="2026-11-01 23:18:22.227815697 +0000 UTC m=+3267.655693627" Nov 01 23:18:22.244223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.244177 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" podStartSLOduration=1.680486371 podStartE2EDuration="2.244164667s" totalImagesPullingTime="563.678296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:22.242952577 +0000 UTC m=+3267.670830513" watchObservedRunningTime="2026-11-01 23:18:22.244164667 +0000 UTC m=+3267.672042597" Nov 01 23:18:22.675436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:22.675401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:23.354999 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:23.354966 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:23.999604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:23.999547 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:24.566413 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:24.566383 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:25.212859 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:25.212827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:25.849632 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:25.849549 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:26.548319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:26.548288 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:27.299949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:27.299918 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:28.064207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:28.064163 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:28.221293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:28.221259 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:18:28.679611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:28.679564 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:29.418882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:29.418844 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:30.171778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:30.171747 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:30.802857 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:30.802826 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:31.532181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:31.532151 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:32.058370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.058295 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:32.269528 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.269498 4981 generic.go:356] "Generic (PLEG): container finished" podID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerID="72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136" exitCode=0 Nov 01 23:18:32.269736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.269612 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" event={"ID":"c5aa6d74-fa20-4203-935a-60be18072fa3","Type":"ContainerDied","Data":"72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136"} Nov 01 23:18:32.270008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.269993 4981 scope.go:122] "RemoveContainer" containerID="72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136" Nov 01 23:18:32.271137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.271108 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:18:32.271259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.271190 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:18:32.271259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.271236 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:18:32.748901 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:32.748868 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:33.198657 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:33.189680 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:33.218630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:33.218599 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:18:33.301440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:33.301399 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" event={"ID":"c5aa6d74-fa20-4203-935a-60be18072fa3","Type":"ContainerStarted","Data":"f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2"} Nov 01 23:18:33.500081 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:33.499986 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:33.636786 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:33.636740 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:33.637952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:33.637926 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" start-of-body= Nov 01 23:18:33.638085 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:33.637975 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" Nov 01 23:18:34.058216 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:34.058178 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:34.264027 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:34.263999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:34.637725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:34.637687 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" start-of-body= Nov 01 23:18:34.637936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:34.637747 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" Nov 01 23:18:34.800078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:34.800039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:34.915459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:34.915375 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:35.020050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:35.020015 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:35.120000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:35.119969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:35.604084 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:35.604051 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:35.637016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:35.636986 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" start-of-body= Nov 01 23:18:35.637200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:35.637033 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" Nov 01 23:18:36.267170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:36.267139 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:36.637550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:36.637464 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" start-of-body= Nov 01 23:18:36.637742 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:36.637513 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" Nov 01 23:18:37.032011 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.031975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:37.566417 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.566385 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:37.637473 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.637439 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" start-of-body= Nov 01 23:18:37.637710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.637499 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.140:15021/healthz/ready\": dial tcp 10.134.1.140:15021: connect: connection refused" Nov 01 23:18:37.825758 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.825673 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx"] Nov 01 23:18:37.828599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.828564 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:37.841309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.841283 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aaf3e1\"/\"openshift-service-ca.crt\"" Nov 01 23:18:37.841671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.841335 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aaf3e1\"/\"simulator-single-main-aaf3e1-kserve-self-signed-certs\"" Nov 01 23:18:37.842024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.841781 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aaf3e1\"/\"kube-root-ca.crt\"" Nov 01 23:18:37.842162 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.842046 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aaf3e1\"/\"default-dockercfg-8rzlr\"" Nov 01 23:18:37.843436 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.843407 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx"] Nov 01 23:18:37.900338 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.900292 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:37.944725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.944692 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-home\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:37.944904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.944735 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvp8d\" (UniqueName: \"kubernetes.io/projected/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-kube-api-access-pvp8d\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:37.944904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.944754 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-model-cache\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:37.945043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.944896 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tls-certs\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:37.945043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.944954 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-dshm\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:37.945159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:37.945070 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tmp-dir\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.045919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.045882 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tls-certs\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.045950 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-dshm\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.045983 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tmp-dir\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.046015 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-home\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.046037 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp8d\" (UniqueName: \"kubernetes.io/projected/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-kube-api-access-pvp8d\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.046119 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-model-cache\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.046498 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-home\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.046530 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-model-cache\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.046736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.046712 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tmp-dir\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.049016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.048982 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-dshm\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.049225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.049209 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tls-certs\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.054183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.054160 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvp8d\" (UniqueName: \"kubernetes.io/projected/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-kube-api-access-pvp8d\") pod \"simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.147506 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.147413 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:38.271832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.271788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:38.423493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.423465 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx"] Nov 01 23:18:38.424926 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:18:38.424900 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5cfd4c7_46a3_4d6e_b475_7de960d9f58a.slice/crio-9fe1c1137798de89eaa39b8ec9bdc73af9836b0635abe0a3115552f80af65238 WatchSource:0}: Error finding container 9fe1c1137798de89eaa39b8ec9bdc73af9836b0635abe0a3115552f80af65238: Status 404 returned error can't find the container with id 9fe1c1137798de89eaa39b8ec9bdc73af9836b0635abe0a3115552f80af65238 Nov 01 23:18:38.641620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.641592 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:38.642083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.642040 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:38.642734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.642710 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:18:38.867642 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:38.867606 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:39.338917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:39.338884 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" event={"ID":"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a","Type":"ContainerStarted","Data":"e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044"} Nov 01 23:18:39.339137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:39.338927 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" event={"ID":"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a","Type":"ContainerStarted","Data":"9fe1c1137798de89eaa39b8ec9bdc73af9836b0635abe0a3115552f80af65238"} Nov 01 23:18:39.357045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:39.356927 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" podStartSLOduration=2.35691243 podStartE2EDuration="2.35691243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:39.356057088 +0000 UTC m=+3284.783935020" watchObservedRunningTime="2026-11-01 23:18:39.35691243 +0000 UTC m=+3284.784790359" Nov 01 23:18:39.621594 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:39.621467 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:40.148323 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:40.148290 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:40.570423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:40.570382 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:41.263982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:41.263946 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:41.865812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:41.865740 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:42.458729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:42.458697 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:43.063417 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:43.063382 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:43.750007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:43.749973 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:44.362367 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:44.362334 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:44.523768 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:18:44.523733 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:18:44.975893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:44.975858 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:45.643363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:45.643330 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:46.263395 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:46.263362 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:46.960407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:46.960378 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:47.613855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:47.613814 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:48.147965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:48.147931 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:48.160489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:48.160459 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:48.344601 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:48.344523 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:48.401501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:48.401421 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:18:48.953340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:48.953305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:49.569938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:49.569906 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:50.296268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:50.296227 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:50.968732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:50.968705 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:51.660801 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:51.660769 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:52.355677 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:52.355647 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:53.052646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:53.052612 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:53.789109 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:53.789076 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:54.119632 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:54.119526 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:54.599175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:54.599145 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:55.255256 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:55.255222 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:55.872817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:55.872783 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:56.654963 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:56.654930 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:57.273886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:57.273853 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:57.868221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:57.868194 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:58.475835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:58.475806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:59.160330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:59.160282 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:18:59.759564 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:18:59.759516 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:00.395037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:00.395003 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:01.035683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:01.035653 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:01.664992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:01.664954 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:02.371127 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:02.371095 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:03.179450 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:03.179416 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:03.848783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:03.848750 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:04.165643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:04.165547 4981 scope.go:122] "RemoveContainer" containerID="a86eb3dabd0188c921adb7e307968540967aefffc7bf6b1699df93a9e69fdac7" Nov 01 23:19:04.174873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:04.174850 4981 scope.go:122] "RemoveContainer" containerID="2ca857f423cd9e4d7f5509c8cecbf5638c3e08bdc9fd433535c471a96ffbb8db" Nov 01 23:19:04.183047 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:04.183026 4981 scope.go:122] "RemoveContainer" containerID="4692cd6c3a4c2ae6876f019287e916f6e961199fd9fce38d5b1f12b0f0e91237" Nov 01 23:19:04.190729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:04.190711 4981 scope.go:122] "RemoveContainer" containerID="1301923a48f4adfe25f7595662b83353a4b706f251d69c7eb74a4018d56b0ffc" Nov 01 23:19:04.198872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:04.198851 4981 scope.go:122] "RemoveContainer" containerID="50715c1b1cb5e3b3e7f9e6e5c44daa7cab9325b3093f8cedaf15dfd6f1d66671" Nov 01 23:19:04.206311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:04.206294 4981 scope.go:122] "RemoveContainer" containerID="2e8d39f5e10c5cd4e03e7db875042b4d436bbb6ad7a93a373bc692ca7958ef46" Nov 01 23:19:04.459916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:04.459883 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:05.151992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:05.151964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:05.753565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:05.753533 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:06.372864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:06.372832 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:06.963520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:06.963486 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:07.578826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:07.578793 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:08.237688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:08.237651 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:08.846982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:08.846950 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:09.447053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:09.447005 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:10.154697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:10.154665 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:10.754536 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:10.754505 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:11.422096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:11.422058 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:12.061418 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:12.061388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:12.652930 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:12.652892 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:13.363914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:13.363884 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:14.063097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:14.063065 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:14.523726 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:19:14.523691 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:19:14.656169 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:14.656094 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:15.377172 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:15.377096 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:15.867244 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:15.867201 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:15.966280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:15.966252 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:16.230988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:16.230934 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:16.613091 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:16.613007 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:16.781405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:16.781375 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:17.511312 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:17.511278 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:18.162082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:18.162047 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:18.971470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:18.971441 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:19.555089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:19.555051 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:20.248850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:20.248813 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:20.937760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:20.937723 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:21.558854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:21.558808 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:21.753308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:21.753278 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx"] Nov 01 23:19:21.848874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:21.848765 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:22.156902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:22.156822 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:22.870427 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:22.870395 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:23.540271 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:23.540237 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:24.158689 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:24.158648 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:24.767140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:24.767110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:25.375892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:25.375858 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:26.010325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:26.010292 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:26.691396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:26.691367 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:27.038883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.038796 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk"] Nov 01 23:19:27.110633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.110597 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:27.167945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.167908 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm"] Nov 01 23:19:27.168193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.168165 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" podUID="7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" containerName="maas-api" containerID="cri-o://9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd" gracePeriod=30 Nov 01 23:19:27.207753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.207717 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:27.288724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.288696 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:27.435812 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.435786 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:19:27.522199 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.522170 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-openshift-service-ca\") pod \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " Nov 01 23:19:27.522376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.522278 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-metrics-certs\") pod \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " Nov 01 23:19:27.522376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.522332 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-maas-api-tls\") pod \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " Nov 01 23:19:27.522467 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.522391 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-kube-api-access-87kt9\" (UniqueName: \"kubernetes.io/projected/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-kube-api-access-87kt9\") pod \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\" (UID: \"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8\") " Nov 01 23:19:27.522552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.522527 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-openshift-service-ca" pod "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" (UID: "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:19:27.522769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.522738 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:27.524468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.524437 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-kube-api-access-87kt9" pod "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" (UID: "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8"). InnerVolumeSpecName "kube-api-access-87kt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:27.524468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.524453 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-metrics-certs" pod "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" (UID: "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:19:27.524668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.524469 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-maas-api-tls" pod "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" (UID: "7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:19:27.603736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.603657 4981 generic.go:356] "Generic (PLEG): container finished" podID="7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" containerID="9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd" exitCode=0 Nov 01 23:19:27.603736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.603701 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" Nov 01 23:19:27.603988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.603742 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" event={"ID":"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8","Type":"ContainerDied","Data":"9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd"} Nov 01 23:19:27.603988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.603800 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm" event={"ID":"7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8","Type":"ContainerDied","Data":"95c71a130abb76cb8877830f3863edea871035c70afc3d22ea74245502ac7d81"} Nov 01 23:19:27.603988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.603829 4981 scope.go:122] "RemoveContainer" containerID="9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd" Nov 01 23:19:27.613385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.613370 4981 scope.go:122] "RemoveContainer" containerID="9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd" Nov 01 23:19:27.613651 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:19:27.613616 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd\": container with ID starting with 9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd not found: ID does not exist" containerID="9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd" Nov 01 23:19:27.613734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.613657 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd"} err="failed to get container status \"9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd\": rpc error: code = NotFound desc = could not find container \"9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd\": container with ID starting with 9363ded7b148daa3c5f1b671efc4615ae16577d6382284ad4155d9a44be7bfbd not found: ID does not exist" Nov 01 23:19:27.624075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.624050 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:27.624075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.624072 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:27.624238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.624084 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87kt9\" (UniqueName: \"kubernetes.io/projected/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8-kube-api-access-87kt9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:27.625408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.625388 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm"] Nov 01 23:19:27.630508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.630489 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-aaf3e1-794568b7fc-sb5dm"] Nov 01 23:19:27.854042 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.853964 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:27.955648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:27.955613 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:28.351913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:28.351882 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:28.732881 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:28.732853 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" path="/var/lib/kubelet/pods/7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8/volumes" Nov 01 23:19:32.039165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:32.039123 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" podUID="ff54d54a-efce-4311-81e7-ccaf573d7487" containerName="payload-processing" containerID="cri-o://c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b" gracePeriod=25 Nov 01 23:19:33.216937 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:33.216896 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" podUID="ff54d54a-efce-4311-81e7-ccaf573d7487" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:19:34.402550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:34.402506 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:34.482567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:34.482538 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:34.696889 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:34.696858 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj"] Nov 01 23:19:34.697096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:34.697075 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" containerID="cri-o://f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2" gracePeriod=30 Nov 01 23:19:34.730871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:34.730840 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:35.874831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.874788 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb"] Nov 01 23:19:35.880171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.880141 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vklvb"] Nov 01 23:19:35.915264 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.915236 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw"] Nov 01 23:19:35.915807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.915788 4981 cpu_manager.go:414] "containerMap: removing container" podUID="7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" containerName="maas-api" Nov 01 23:19:35.915923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.915809 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" containerName="maas-api" Nov 01 23:19:35.915987 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.915945 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebc7c70-d6d9-49b5-9bfa-f66addaaa0b8" containerName="maas-api" Nov 01 23:19:35.920296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.920276 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:35.923832 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.923815 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:19:35.924097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.924035 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:19:35.924902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.924881 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:19:35.925037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.925020 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:19:35.928000 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:35.927974 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw"] Nov 01 23:19:36.003265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.003231 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.003265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.003267 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cj5\" (UniqueName: \"kubernetes.io/projected/00f7747a-8503-47ae-90a5-4b61394d9af5-kube-api-access-s7cj5\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.003494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.003427 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.105008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.104978 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.105008 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.105010 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cj5\" (UniqueName: \"kubernetes.io/projected/00f7747a-8503-47ae-90a5-4b61394d9af5-kube-api-access-s7cj5\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.105230 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.105105 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.105660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.105624 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.107403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.107383 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.113051 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.113028 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cj5\" (UniqueName: \"kubernetes.io/projected/00f7747a-8503-47ae-90a5-4b61394d9af5-kube-api-access-s7cj5\") pod \"maas-api-revoke-sub-3d4156f1-d87mw\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.235878 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.235853 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:36.506926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.506901 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw"] Nov 01 23:19:36.509360 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:19:36.509328 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f7747a_8503_47ae_90a5_4b61394d9af5.slice/crio-f8d2e7139e6fd549515f07f70dfb1d726b393df059f4287dc103d174a8a12a76 WatchSource:0}: Error finding container f8d2e7139e6fd549515f07f70dfb1d726b393df059f4287dc103d174a8a12a76: Status 404 returned error can't find the container with id f8d2e7139e6fd549515f07f70dfb1d726b393df059f4287dc103d174a8a12a76 Nov 01 23:19:36.654121 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.654084 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" event={"ID":"00f7747a-8503-47ae-90a5-4b61394d9af5","Type":"ContainerStarted","Data":"f8d2e7139e6fd549515f07f70dfb1d726b393df059f4287dc103d174a8a12a76"} Nov 01 23:19:36.733088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.733047 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6876a52-ba28-469f-8a0b-863571c5db71" path="/var/lib/kubelet/pods/a6876a52-ba28-469f-8a0b-863571c5db71/volumes" Nov 01 23:19:36.773701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:36.773565 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" podUID="e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" containerName="main" containerID="cri-o://e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044" gracePeriod=45 Nov 01 23:19:37.022829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.022092 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:19:37.116357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.116258 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-dshm\") pod \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " Nov 01 23:19:37.116357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.116321 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-home\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-home\") pod \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " Nov 01 23:19:37.116616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.116383 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tmp-dir\") pod \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " Nov 01 23:19:37.116616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.116474 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-kube-api-access-pvp8d\" (UniqueName: \"kubernetes.io/projected/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-kube-api-access-pvp8d\") pod \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " Nov 01 23:19:37.116616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.116530 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-model-cache\") pod \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " Nov 01 23:19:37.116784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.116632 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tls-certs\") pod \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\" (UID: \"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a\") " Nov 01 23:19:37.117522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.117492 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tmp-dir" pod "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" (UID: "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:37.117675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.117621 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-home" pod "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" (UID: "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:37.117746 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.117701 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-model-cache" pod "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" (UID: "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:37.120094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.120061 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-kube-api-access-pvp8d" pod "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" (UID: "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a"). InnerVolumeSpecName "kube-api-access-pvp8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:37.120284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.120263 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-dshm" pod "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" (UID: "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:37.120343 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.120290 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tls-certs" pod "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" (UID: "e5cfd4c7-46a3-4d6e-b475-7de960d9f58a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:19:37.218101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.218066 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvp8d\" (UniqueName: \"kubernetes.io/projected/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-kube-api-access-pvp8d\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:37.218101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.218097 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:37.218101 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.218108 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:37.218351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.218117 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:37.218351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.218125 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:37.218351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.218134 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:37.660805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.660764 4981 generic.go:356] "Generic (PLEG): container finished" podID="e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" containerID="e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044" exitCode=0 Nov 01 23:19:37.660805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.660809 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" Nov 01 23:19:37.661098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.660798 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" event={"ID":"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a","Type":"ContainerDied","Data":"e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044"} Nov 01 23:19:37.661098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.660929 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx" event={"ID":"e5cfd4c7-46a3-4d6e-b475-7de960d9f58a","Type":"ContainerDied","Data":"9fe1c1137798de89eaa39b8ec9bdc73af9836b0635abe0a3115552f80af65238"} Nov 01 23:19:37.661098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.660954 4981 scope.go:122] "RemoveContainer" containerID="e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044" Nov 01 23:19:37.662709 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.662684 4981 generic.go:356] "Generic (PLEG): container finished" podID="00f7747a-8503-47ae-90a5-4b61394d9af5" containerID="3cbbdc31cd59e41c09048419ffaccb089e6a7ee321bde88b24e6019812dcf4c7" exitCode=0 Nov 01 23:19:37.662829 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.662732 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" event={"ID":"00f7747a-8503-47ae-90a5-4b61394d9af5","Type":"ContainerDied","Data":"3cbbdc31cd59e41c09048419ffaccb089e6a7ee321bde88b24e6019812dcf4c7"} Nov 01 23:19:37.672656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.672640 4981 scope.go:122] "RemoveContainer" containerID="e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044" Nov 01 23:19:37.672982 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:19:37.672956 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044\": container with ID starting with e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044 not found: ID does not exist" containerID="e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044" Nov 01 23:19:37.673103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.672991 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044"} err="failed to get container status \"e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044\": rpc error: code = NotFound desc = could not find container \"e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044\": container with ID starting with e86077c695be040d51d797f75f9e46d46c83e1e850f93fcf1b2a3214a9e39044 not found: ID does not exist" Nov 01 23:19:37.697247 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.697222 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx"] Nov 01 23:19:37.700038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:37.700016 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-aaf3e1/simulator-single-main-aaf3e1-kserve-7cbd9f787c-nh4hx"] Nov 01 23:19:38.643875 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.643828 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:19:38.644260 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.643906 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:19:38.734460 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.734426 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" path="/var/lib/kubelet/pods/e5cfd4c7-46a3-4d6e-b475-7de960d9f58a/volumes" Nov 01 23:19:38.806581 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.806554 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:38.936835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.936808 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-xks-ca\") pod \"00f7747a-8503-47ae-90a5-4b61394d9af5\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " Nov 01 23:19:38.937002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.936851 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-openshift-ca\") pod \"00f7747a-8503-47ae-90a5-4b61394d9af5\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " Nov 01 23:19:38.937002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.936904 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f7747a-8503-47ae-90a5-4b61394d9af5-kube-api-access-s7cj5\" (UniqueName: \"kubernetes.io/projected/00f7747a-8503-47ae-90a5-4b61394d9af5-kube-api-access-s7cj5\") pod \"00f7747a-8503-47ae-90a5-4b61394d9af5\" (UID: \"00f7747a-8503-47ae-90a5-4b61394d9af5\") " Nov 01 23:19:38.937327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.937300 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-openshift-ca" pod "00f7747a-8503-47ae-90a5-4b61394d9af5" (UID: "00f7747a-8503-47ae-90a5-4b61394d9af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:19:38.939068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.939043 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-xks-ca" pod "00f7747a-8503-47ae-90a5-4b61394d9af5" (UID: "00f7747a-8503-47ae-90a5-4b61394d9af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:19:38.939068 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:38.939056 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f7747a-8503-47ae-90a5-4b61394d9af5-kube-api-access-s7cj5" pod "00f7747a-8503-47ae-90a5-4b61394d9af5" (UID: "00f7747a-8503-47ae-90a5-4b61394d9af5"). InnerVolumeSpecName "kube-api-access-s7cj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:39.038065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:39.038025 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:39.038065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:39.038059 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f7747a-8503-47ae-90a5-4b61394d9af5-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:39.038282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:39.038073 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7cj5\" (UniqueName: \"kubernetes.io/projected/00f7747a-8503-47ae-90a5-4b61394d9af5-kube-api-access-s7cj5\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:39.674828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:39.674796 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" Nov 01 23:19:39.675279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:39.674848 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw" event={"ID":"00f7747a-8503-47ae-90a5-4b61394d9af5","Type":"ContainerDied","Data":"f8d2e7139e6fd549515f07f70dfb1d726b393df059f4287dc103d174a8a12a76"} Nov 01 23:19:39.675279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:39.674905 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8d2e7139e6fd549515f07f70dfb1d726b393df059f4287dc103d174a8a12a76" Nov 01 23:19:39.950184 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:39.950160 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:19:40.048166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048130 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-token\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-token\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048234 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-credential-socket\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048286 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-podinfo\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048318 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-socket\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048364 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5aa6d74-fa20-4203-935a-60be18072fa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5aa6d74-fa20-4203-935a-60be18072fa3-istiod-ca-cert\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048410 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-envoy\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048439 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-kube-api-access-n58tg\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-kube-api-access-n58tg\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048521 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-data\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048552 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-certs\") pod \"c5aa6d74-fa20-4203-935a-60be18072fa3\" (UID: \"c5aa6d74-fa20-4203-935a-60be18072fa3\") " Nov 01 23:19:40.048727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048610 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-credential-socket" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:40.048727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048680 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-socket" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:40.048961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048835 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-data" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:40.048961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048896 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-certs" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:40.048961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.048821 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aa6d74-fa20-4203-935a-60be18072fa3-istiod-ca-cert" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:19:40.049116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.049028 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.049116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.049047 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.049116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.049062 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5aa6d74-fa20-4203-935a-60be18072fa3-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.049116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.049076 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.049116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.049091 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.050600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.050547 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-token" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:40.050741 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.050610 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-kube-api-access-n58tg" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "kube-api-access-n58tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:40.050944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.050923 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-podinfo" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:19:40.051053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.050969 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-envoy" pod "c5aa6d74-fa20-4203-935a-60be18072fa3" (UID: "c5aa6d74-fa20-4203-935a-60be18072fa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:19:40.150465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.150436 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.150465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.150459 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.150465 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.150471 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n58tg\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-kube-api-access-n58tg\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.150752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.150480 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5aa6d74-fa20-4203-935a-60be18072fa3-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:40.682179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.682144 4981 generic.go:356] "Generic (PLEG): container finished" podID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerID="f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2" exitCode=0 Nov 01 23:19:40.682714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.682205 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" event={"ID":"c5aa6d74-fa20-4203-935a-60be18072fa3","Type":"ContainerDied","Data":"f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2"} Nov 01 23:19:40.682714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.682207 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" Nov 01 23:19:40.682714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.682245 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj" event={"ID":"c5aa6d74-fa20-4203-935a-60be18072fa3","Type":"ContainerDied","Data":"cb71ed33b25d2c74460f52a08e49b631591e2bf631c3346e415497ef89389db3"} Nov 01 23:19:40.682714 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.682267 4981 scope.go:122] "RemoveContainer" containerID="f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2" Nov 01 23:19:40.692509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.692487 4981 scope.go:122] "RemoveContainer" containerID="72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136" Nov 01 23:19:40.701219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.701199 4981 scope.go:122] "RemoveContainer" containerID="f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2" Nov 01 23:19:40.701495 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:19:40.701474 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2\": container with ID starting with f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2 not found: ID does not exist" containerID="f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2" Nov 01 23:19:40.701541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.701503 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2"} err="failed to get container status \"f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2\": rpc error: code = NotFound desc = could not find container \"f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2\": container with ID starting with f188152089105d641bf72d08c0d0120087f69a1ef5ca906b19e8866f04c88da2 not found: ID does not exist" Nov 01 23:19:40.701541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.701521 4981 scope.go:122] "RemoveContainer" containerID="72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136" Nov 01 23:19:40.701775 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:19:40.701756 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136\": container with ID starting with 72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136 not found: ID does not exist" containerID="72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136" Nov 01 23:19:40.701835 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.701782 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136"} err="failed to get container status \"72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136\": rpc error: code = NotFound desc = could not find container \"72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136\": container with ID starting with 72391616a933dd7d230fe0d72bd956a8b483054ee45f3d6fce63e7dff17b7136 not found: ID does not exist" Nov 01 23:19:40.716726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.716697 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj"] Nov 01 23:19:40.723031 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.723011 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aaf3e1-openshift-default-5bdc7766699dwjj"] Nov 01 23:19:40.732956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:40.732936 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" path="/var/lib/kubelet/pods/c5aa6d74-fa20-4203-935a-60be18072fa3/volumes" Nov 01 23:19:43.217064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:43.217030 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" podUID="ff54d54a-efce-4311-81e7-ccaf573d7487" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 23:19:44.523844 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:19:44.523803 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:19:48.050673 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:48.050636 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9"] Nov 01 23:19:48.054286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:48.054255 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-ggsv9"] Nov 01 23:19:48.751310 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:48.751264 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05" path="/var/lib/kubelet/pods/6e466cd3-9a3a-43c3-84a2-1ccf81ca0a05/volumes" Nov 01 23:19:52.299146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.299070 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:19:52.392125 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.392091 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff54d54a-efce-4311-81e7-ccaf573d7487-kube-api-access-jnwhd\" (UniqueName: \"kubernetes.io/projected/ff54d54a-efce-4311-81e7-ccaf573d7487-kube-api-access-jnwhd\") pod \"ff54d54a-efce-4311-81e7-ccaf573d7487\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " Nov 01 23:19:52.392305 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.392156 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff54d54a-efce-4311-81e7-ccaf573d7487-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff54d54a-efce-4311-81e7-ccaf573d7487-plugins-config-volume\") pod \"ff54d54a-efce-4311-81e7-ccaf573d7487\" (UID: \"ff54d54a-efce-4311-81e7-ccaf573d7487\") " Nov 01 23:19:52.392522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.392494 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff54d54a-efce-4311-81e7-ccaf573d7487-plugins-config-volume" pod "ff54d54a-efce-4311-81e7-ccaf573d7487" (UID: "ff54d54a-efce-4311-81e7-ccaf573d7487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:19:52.394202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.394181 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff54d54a-efce-4311-81e7-ccaf573d7487-kube-api-access-jnwhd" pod "ff54d54a-efce-4311-81e7-ccaf573d7487" (UID: "ff54d54a-efce-4311-81e7-ccaf573d7487"). InnerVolumeSpecName "kube-api-access-jnwhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:19:52.493041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.493009 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwhd\" (UniqueName: \"kubernetes.io/projected/ff54d54a-efce-4311-81e7-ccaf573d7487-kube-api-access-jnwhd\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:52.493041 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.493037 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff54d54a-efce-4311-81e7-ccaf573d7487-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:19:52.763924 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.763887 4981 generic.go:356] "Generic (PLEG): container finished" podID="ff54d54a-efce-4311-81e7-ccaf573d7487" containerID="c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b" exitCode=0 Nov 01 23:19:52.764098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.763928 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" Nov 01 23:19:52.764098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.763969 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" event={"ID":"ff54d54a-efce-4311-81e7-ccaf573d7487","Type":"ContainerDied","Data":"c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b"} Nov 01 23:19:52.764098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.764027 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk" event={"ID":"ff54d54a-efce-4311-81e7-ccaf573d7487","Type":"ContainerDied","Data":"e4f3e9dddfc48bb04b94267696925dcb709eba6ba21d31472207ede95240423d"} Nov 01 23:19:52.764098 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.764055 4981 scope.go:122] "RemoveContainer" containerID="c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b" Nov 01 23:19:52.772872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.772851 4981 scope.go:122] "RemoveContainer" containerID="c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b" Nov 01 23:19:52.773146 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:19:52.773130 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b\": container with ID starting with c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b not found: ID does not exist" containerID="c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b" Nov 01 23:19:52.773221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.773152 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b"} err="failed to get container status \"c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b\": rpc error: code = NotFound desc = could not find container \"c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b\": container with ID starting with c686f7fe15cf844cf739a39dda64bded94aa6fb92bf19fad9fc9aacf2f947a4b not found: ID does not exist" Nov 01 23:19:52.781398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.781376 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk"] Nov 01 23:19:52.789688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:52.789669 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aaf3e1-6d4d8b857rprsk"] Nov 01 23:19:54.734635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:54.734605 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff54d54a-efce-4311-81e7-ccaf573d7487" path="/var/lib/kubelet/pods/ff54d54a-efce-4311-81e7-ccaf573d7487/volumes" Nov 01 23:19:57.988031 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:57.987999 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:19:58.898219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:19:58.898183 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:20:00.893783 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:00.893741 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:20:04.475171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:04.475133 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:20:14.523409 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:20:14.523367 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:20:15.907514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:15.907475 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:20:16.283872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:16.283836 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:20:25.020252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:25.020218 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6"] Nov 01 23:20:25.023955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:25.023931 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-27kz6"] Nov 01 23:20:26.741238 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:26.741203 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b83401-d56c-47fe-b856-43187a10d73b" path="/var/lib/kubelet/pods/f1b83401-d56c-47fe-b856-43187a10d73b/volumes" Nov 01 23:20:44.523756 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:20:44.523712 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:20:45.020620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:45.020566 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s"] Nov 01 23:20:45.022390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:45.022363 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-gw74s"] Nov 01 23:20:46.732888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:20:46.732856 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b87954-c27e-46d2-9ad2-7ed126e5f993" path="/var/lib/kubelet/pods/53b87954-c27e-46d2-9ad2-7ed126e5f993/volumes" Nov 01 23:21:03.022985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:03.022944 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz"] Nov 01 23:21:03.026676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:03.026642 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t6wwz"] Nov 01 23:21:04.269877 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:04.269819 4981 scope.go:122] "RemoveContainer" containerID="3fc43f7309514e4d4c5ad8ca0d42eb4014b7431486a7fb0bc85cafc2fbb5e492" Nov 01 23:21:04.732948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:04.732915 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc104d44-8de8-4737-9016-48ea845edee5" path="/var/lib/kubelet/pods/fc104d44-8de8-4737-9016-48ea845edee5/volumes" Nov 01 23:21:14.530015 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:21:14.529972 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:21:16.003597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:16.003538 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:16.417621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:16.417513 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:17.645766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.645733 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l"] Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646275 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646292 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646302 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646308 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646316 4981 cpu_manager.go:414] "containerMap: removing container" podUID="ff54d54a-efce-4311-81e7-ccaf573d7487" containerName="payload-processing" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646322 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff54d54a-efce-4311-81e7-ccaf573d7487" containerName="payload-processing" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646338 4981 cpu_manager.go:414] "containerMap: removing container" podUID="00f7747a-8503-47ae-90a5-4b61394d9af5" containerName="revoke-keys" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646343 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f7747a-8503-47ae-90a5-4b61394d9af5" containerName="revoke-keys" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646351 4981 cpu_manager.go:414] "containerMap: removing container" podUID="e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" containerName="main" Nov 01 23:21:17.646385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646356 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" containerName="main" Nov 01 23:21:17.646779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646435 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" Nov 01 23:21:17.646779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646445 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5aa6d74-fa20-4203-935a-60be18072fa3" containerName="istio-proxy" Nov 01 23:21:17.646779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646450 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff54d54a-efce-4311-81e7-ccaf573d7487" containerName="payload-processing" Nov 01 23:21:17.646779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646458 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f7747a-8503-47ae-90a5-4b61394d9af5" containerName="revoke-keys" Nov 01 23:21:17.646779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.646463 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cfd4c7-46a3-4d6e-b475-7de960d9f58a" containerName="main" Nov 01 23:21:17.650854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.650839 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.655597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.655523 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:21:17.655597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.655598 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:21:17.656095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.655926 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:21:17.656095 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.656061 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:21:17.656748 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.656727 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l"] Nov 01 23:21:17.748945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.748910 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.749139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.748951 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.749187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.749156 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkq4s\" (UniqueName: \"kubernetes.io/projected/c7a55ba9-05da-43bd-b577-27f7f346992b-kube-api-access-jkq4s\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.850002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.849965 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.850150 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.850132 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkq4s\" (UniqueName: \"kubernetes.io/projected/c7a55ba9-05da-43bd-b577-27f7f346992b-kube-api-access-jkq4s\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.850202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.850176 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.850784 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.850764 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.852544 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.852521 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.857970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.857946 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkq4s\" (UniqueName: \"kubernetes.io/projected/c7a55ba9-05da-43bd-b577-27f7f346992b-kube-api-access-jkq4s\") pod \"maas-api-revoke-sub-d576e68b-cwd5l\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:17.966344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:17.966311 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:18.244420 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:18.244396 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l"] Nov 01 23:21:18.246828 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:21:18.246792 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a55ba9_05da_43bd_b577_27f7f346992b.slice/crio-d5e49748677e11ae360514cf0561749084769ef0d3a629a157d5f5b521c91668 WatchSource:0}: Error finding container d5e49748677e11ae360514cf0561749084769ef0d3a629a157d5f5b521c91668: Status 404 returned error can't find the container with id d5e49748677e11ae360514cf0561749084769ef0d3a629a157d5f5b521c91668 Nov 01 23:21:19.256474 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:19.256383 4981 generic.go:356] "Generic (PLEG): container finished" podID="c7a55ba9-05da-43bd-b577-27f7f346992b" containerID="eabc21de3c7c2d676d0aa6dc5fed6a41ae3f685bfd63da780681b6b91edcf565" exitCode=0 Nov 01 23:21:19.256994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:19.256478 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" event={"ID":"c7a55ba9-05da-43bd-b577-27f7f346992b","Type":"ContainerDied","Data":"eabc21de3c7c2d676d0aa6dc5fed6a41ae3f685bfd63da780681b6b91edcf565"} Nov 01 23:21:19.256994 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:19.256557 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" event={"ID":"c7a55ba9-05da-43bd-b577-27f7f346992b","Type":"ContainerStarted","Data":"d5e49748677e11ae360514cf0561749084769ef0d3a629a157d5f5b521c91668"} Nov 01 23:21:20.395660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.395632 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:20.474490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.474452 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-xks-ca\") pod \"c7a55ba9-05da-43bd-b577-27f7f346992b\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " Nov 01 23:21:20.474727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.474555 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-openshift-ca\") pod \"c7a55ba9-05da-43bd-b577-27f7f346992b\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " Nov 01 23:21:20.474727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.474610 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a55ba9-05da-43bd-b577-27f7f346992b-kube-api-access-jkq4s\" (UniqueName: \"kubernetes.io/projected/c7a55ba9-05da-43bd-b577-27f7f346992b-kube-api-access-jkq4s\") pod \"c7a55ba9-05da-43bd-b577-27f7f346992b\" (UID: \"c7a55ba9-05da-43bd-b577-27f7f346992b\") " Nov 01 23:21:20.474943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.474915 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-openshift-ca" pod "c7a55ba9-05da-43bd-b577-27f7f346992b" (UID: "c7a55ba9-05da-43bd-b577-27f7f346992b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:21:20.476734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.476711 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-xks-ca" pod "c7a55ba9-05da-43bd-b577-27f7f346992b" (UID: "c7a55ba9-05da-43bd-b577-27f7f346992b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:21:20.476817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.476724 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a55ba9-05da-43bd-b577-27f7f346992b-kube-api-access-jkq4s" pod "c7a55ba9-05da-43bd-b577-27f7f346992b" (UID: "c7a55ba9-05da-43bd-b577-27f7f346992b"). InnerVolumeSpecName "kube-api-access-jkq4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:21:20.576173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.576083 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:21:20.576173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.576114 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkq4s\" (UniqueName: \"kubernetes.io/projected/c7a55ba9-05da-43bd-b577-27f7f346992b-kube-api-access-jkq4s\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:21:20.576173 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:20.576126 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a55ba9-05da-43bd-b577-27f7f346992b-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:21:21.270093 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:21.270052 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" event={"ID":"c7a55ba9-05da-43bd-b577-27f7f346992b","Type":"ContainerDied","Data":"d5e49748677e11ae360514cf0561749084769ef0d3a629a157d5f5b521c91668"} Nov 01 23:21:21.270093 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:21.270086 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l" Nov 01 23:21:21.270315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:21.270144 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5e49748677e11ae360514cf0561749084769ef0d3a629a157d5f5b521c91668" Nov 01 23:21:22.024132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:22.024090 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b"] Nov 01 23:21:22.027015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:22.026987 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xm89b"] Nov 01 23:21:22.733176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:22.733142 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f" path="/var/lib/kubelet/pods/24427c8d-16ac-4bd2-8a4b-b5a0c822ac7f/volumes" Nov 01 23:21:35.296932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.296883 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx"] Nov 01 23:21:35.297393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.297377 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c7a55ba9-05da-43bd-b577-27f7f346992b" containerName="revoke-keys" Nov 01 23:21:35.297445 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.297395 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a55ba9-05da-43bd-b577-27f7f346992b" containerName="revoke-keys" Nov 01 23:21:35.297502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.297492 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a55ba9-05da-43bd-b577-27f7f346992b" containerName="revoke-keys" Nov 01 23:21:35.300699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.300682 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.307208 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.307181 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:21:35.307490 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.307466 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:21:35.307683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.307560 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:21:35.307683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.307668 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:21:35.308923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.308895 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx"] Nov 01 23:21:35.418512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.418477 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.418705 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.418523 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.418762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.418698 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khj97\" (UniqueName: \"kubernetes.io/projected/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-kube-api-access-khj97\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.519649 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.519616 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khj97\" (UniqueName: \"kubernetes.io/projected/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-kube-api-access-khj97\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.519841 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.519673 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.519841 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.519701 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.520364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.520344 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.522069 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.522048 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.527329 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.527302 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khj97\" (UniqueName: \"kubernetes.io/projected/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-kube-api-access-khj97\") pod \"maas-api-revoke-sub-d2bc93bc-jtwxx\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.616752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.616665 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:35.888294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:35.888266 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx"] Nov 01 23:21:35.891070 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:21:35.891040 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a37203f_7ed3_45eb_b9e3_1f51dcdef54d.slice/crio-58a4a2a45fee7c84b461018c137440480b060307a47cf3f0c46b1f4f09845fde WatchSource:0}: Error finding container 58a4a2a45fee7c84b461018c137440480b060307a47cf3f0c46b1f4f09845fde: Status 404 returned error can't find the container with id 58a4a2a45fee7c84b461018c137440480b060307a47cf3f0c46b1f4f09845fde Nov 01 23:21:36.358976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:36.358937 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" event={"ID":"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d","Type":"ContainerStarted","Data":"6cbf8a0638b1fb09e069dad753419fd2b1805ab5bdf1e8650f2a2fcd35eec1f4"} Nov 01 23:21:36.359403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:36.358996 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" event={"ID":"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d","Type":"ContainerStarted","Data":"58a4a2a45fee7c84b461018c137440480b060307a47cf3f0c46b1f4f09845fde"} Nov 01 23:21:36.377166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:36.377118 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" podStartSLOduration=1.377103606 podStartE2EDuration="1.377103606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:21:36.375089607 +0000 UTC m=+3461.802967542" watchObservedRunningTime="2026-11-01 23:21:36.377103606 +0000 UTC m=+3461.804981536" Nov 01 23:21:37.366509 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:37.366471 4981 generic.go:356] "Generic (PLEG): container finished" podID="6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" containerID="6cbf8a0638b1fb09e069dad753419fd2b1805ab5bdf1e8650f2a2fcd35eec1f4" exitCode=0 Nov 01 23:21:37.366944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:37.366557 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" event={"ID":"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d","Type":"ContainerDied","Data":"6cbf8a0638b1fb09e069dad753419fd2b1805ab5bdf1e8650f2a2fcd35eec1f4"} Nov 01 23:21:38.508729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.508699 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:38.547042 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.547010 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-xks-ca\") pod \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " Nov 01 23:21:38.547219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.547105 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-kube-api-access-khj97\" (UniqueName: \"kubernetes.io/projected/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-kube-api-access-khj97\") pod \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " Nov 01 23:21:38.547219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.547173 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-openshift-ca\") pod \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\" (UID: \"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d\") " Nov 01 23:21:38.547541 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.547515 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-openshift-ca" pod "6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" (UID: "6a37203f-7ed3-45eb-b9e3-1f51dcdef54d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:21:38.549403 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.549376 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-kube-api-access-khj97" pod "6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" (UID: "6a37203f-7ed3-45eb-b9e3-1f51dcdef54d"). InnerVolumeSpecName "kube-api-access-khj97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:21:38.549520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.549442 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-xks-ca" pod "6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" (UID: "6a37203f-7ed3-45eb-b9e3-1f51dcdef54d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:21:38.648493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.648424 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:21:38.648493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.648450 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khj97\" (UniqueName: \"kubernetes.io/projected/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-kube-api-access-khj97\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:21:38.648493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:38.648460 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:21:39.020094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:39.020061 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g"] Nov 01 23:21:39.023142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:39.023114 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7296g"] Nov 01 23:21:39.380332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:39.380254 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" Nov 01 23:21:39.380332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:39.380276 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx" event={"ID":"6a37203f-7ed3-45eb-b9e3-1f51dcdef54d","Type":"ContainerDied","Data":"58a4a2a45fee7c84b461018c137440480b060307a47cf3f0c46b1f4f09845fde"} Nov 01 23:21:39.380332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:39.380319 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a4a2a45fee7c84b461018c137440480b060307a47cf3f0c46b1f4f09845fde" Nov 01 23:21:40.732594 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:40.732554 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f5a86e-8126-402f-90dc-e64752437f8e" path="/var/lib/kubelet/pods/b1f5a86e-8126-402f-90dc-e64752437f8e/volumes" Nov 01 23:21:44.522946 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:21:44.522909 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:21:46.892663 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:46.892628 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:47.355202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.355162 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p"] Nov 01 23:21:47.356022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.355996 4981 cpu_manager.go:414] "containerMap: removing container" podUID="6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" containerName="revoke-keys" Nov 01 23:21:47.356022 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.356022 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" containerName="revoke-keys" Nov 01 23:21:47.356183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.356128 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" containerName="revoke-keys" Nov 01 23:21:47.360045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.360023 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.370140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.370120 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-45dd82-openshift-default-dockercfg-8wf5l\"" Nov 01 23:21:47.378388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.378354 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p"] Nov 01 23:21:47.434673 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.434636 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-token\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.434860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.434688 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-podinfo\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.434860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.434717 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-credential-socket\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.434860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.434778 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f0ab30-8028-4576-a5b2-e82c42148ad8-istiod-ca-cert\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.434860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.434802 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-certs\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.434860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.434829 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-envoy\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.435064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.434895 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99f9\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-kube-api-access-q99f9\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.435154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.435114 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-data\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.435303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.435193 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-socket\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.469311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.469284 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:47.535765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.535733 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f0ab30-8028-4576-a5b2-e82c42148ad8-istiod-ca-cert\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.535765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.535768 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-certs\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.535969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.535793 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-envoy\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.535969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.535849 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99f9\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-kube-api-access-q99f9\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.535969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.535883 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-data\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536043 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-socket\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536120 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-token\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536166 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-podinfo\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536261 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536195 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-credential-socket\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536257 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-certs\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536448 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536376 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-data\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536459 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-socket\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536623 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-credential-socket\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.536782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.536630 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f0ab30-8028-4576-a5b2-e82c42148ad8-istiod-ca-cert\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.538227 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.538206 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-envoy\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.538851 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.538830 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-podinfo\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.547170 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.547147 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-token\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.547556 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.547531 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99f9\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-kube-api-access-q99f9\") pod \"e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.589014 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.588975 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:47.673501 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.673471 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:47.956497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.956470 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p"] Nov 01 23:21:47.958738 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:21:47.958705 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f0ab30_8028_4576_a5b2_e82c42148ad8.slice/crio-ecc289ec99a50b020abc680af9d53eab52b0a795275d1513699c550f93f537b0 WatchSource:0}: Error finding container ecc289ec99a50b020abc680af9d53eab52b0a795275d1513699c550f93f537b0: Status 404 returned error can't find the container with id ecc289ec99a50b020abc680af9d53eab52b0a795275d1513699c550f93f537b0 Nov 01 23:21:47.960909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.960876 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:21:47.961012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.960940 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:21:47.961012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:47.960970 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:21:48.429406 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:48.429368 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" event={"ID":"94f0ab30-8028-4576-a5b2-e82c42148ad8","Type":"ContainerStarted","Data":"422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9"} Nov 01 23:21:48.429602 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:48.429429 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" event={"ID":"94f0ab30-8028-4576-a5b2-e82c42148ad8","Type":"ContainerStarted","Data":"ecc289ec99a50b020abc680af9d53eab52b0a795275d1513699c550f93f537b0"} Nov 01 23:21:48.450195 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:48.450148 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podStartSLOduration=1.45013574 podStartE2EDuration="1.45013574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:21:48.44952046 +0000 UTC m=+3473.877398392" watchObservedRunningTime="2026-11-01 23:21:48.45013574 +0000 UTC m=+3473.878013670" Nov 01 23:21:48.673943 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:48.673910 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:49.679026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:49.678998 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:50.441355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:50.441323 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:50.442279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:50.442255 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:21:50.603046 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:50.603009 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:50.701880 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:50.701779 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:51.715726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.715689 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h"] Nov 01 23:21:51.720886 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.720854 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.731644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.731615 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h"] Nov 01 23:21:51.732799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.732271 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 23:21:51.732799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.732714 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:21:51.732799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.732762 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-45dd82-serving-cert\"" Nov 01 23:21:51.732799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.732796 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:21:51.733242 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.733109 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:21:51.775063 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.775008 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:51.874005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.873946 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:51.881820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.881799 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.881974 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.881955 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc23ea8f-0587-4217-b70b-565c6c7716b4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.882035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.881988 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bf5t\" (UniqueName: \"kubernetes.io/projected/bc23ea8f-0587-4217-b70b-565c6c7716b4-kube-api-access-4bf5t\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.882136 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.882113 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.983117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.983022 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.983483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.983451 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.983656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.983566 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc23ea8f-0587-4217-b70b-565c6c7716b4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.983656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.983621 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bf5t\" (UniqueName: \"kubernetes.io/projected/bc23ea8f-0587-4217-b70b-565c6c7716b4-kube-api-access-4bf5t\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.984768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.984565 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:51.984768 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.984700 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc23ea8f-0587-4217-b70b-565c6c7716b4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.986405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.986374 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.986858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.986835 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:51.993496 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:51.993466 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bf5t\" (UniqueName: \"kubernetes.io/projected/bc23ea8f-0587-4217-b70b-565c6c7716b4-kube-api-access-4bf5t\") pod \"maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:52.040808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:52.040765 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:52.322527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:52.322503 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h"] Nov 01 23:21:52.324805 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:21:52.324771 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc23ea8f_0587_4217_b70b_565c6c7716b4.slice/crio-bd4b56512efed8173bb96b915677c0c224e6050fc039af508f1780641f0bb27d WatchSource:0}: Error finding container bd4b56512efed8173bb96b915677c0c224e6050fc039af508f1780641f0bb27d: Status 404 returned error can't find the container with id bd4b56512efed8173bb96b915677c0c224e6050fc039af508f1780641f0bb27d Nov 01 23:21:52.460713 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:52.460671 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" event={"ID":"bc23ea8f-0587-4217-b70b-565c6c7716b4","Type":"ContainerStarted","Data":"bd4b56512efed8173bb96b915677c0c224e6050fc039af508f1780641f0bb27d"} Nov 01 23:21:52.562688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:52.562653 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:53.386414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:53.386383 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:53.467466 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:53.467433 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" event={"ID":"bc23ea8f-0587-4217-b70b-565c6c7716b4","Type":"ContainerStarted","Data":"a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc"} Nov 01 23:21:53.467655 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:53.467503 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:54.184528 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:54.184482 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:54.967234 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:54.967198 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:55.595734 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:55.595694 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:56.299410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:56.299378 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:56.966678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:56.966638 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:57.669111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:57.669066 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:58.266269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:58.266230 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:58.951096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:58.951063 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:21:59.500431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:59.500398 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:21:59.518592 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:59.518519 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" podStartSLOduration=8.072153301 podStartE2EDuration="8.518503706s" totalImagesPullingTime="446.350405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:21:53.487768027 +0000 UTC m=+3478.915645959" watchObservedRunningTime="2026-11-01 23:21:59.518503706 +0000 UTC m=+3484.946381637" Nov 01 23:21:59.548630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:21:59.548597 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:00.147324 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:00.147284 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:00.816940 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:00.816893 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:01.481049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:01.481015 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:02.168890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:02.168852 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:02.956921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:02.956884 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:03.569140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:03.569097 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:04.262018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.261969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:04.291096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.291071 4981 scope.go:122] "RemoveContainer" containerID="f44b340a25db84c9939efef81603763aa8ce2662697544a8058506eab7b16140" Nov 01 23:22:04.299933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.299914 4981 scope.go:122] "RemoveContainer" containerID="93e04419310b8ad071cd37b5005996b2ccbd52651ea481d776272774ee00d9b7" Nov 01 23:22:04.309199 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.309164 4981 scope.go:122] "RemoveContainer" containerID="06655feadd602a2d4d521a8fb25d41455cdde8d3d8f59485d0e36424ec8c45b5" Nov 01 23:22:04.552303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.552216 4981 generic.go:356] "Generic (PLEG): container finished" podID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerID="422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9" exitCode=0 Nov 01 23:22:04.552303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.552251 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" event={"ID":"94f0ab30-8028-4576-a5b2-e82c42148ad8","Type":"ContainerDied","Data":"422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9"} Nov 01 23:22:04.552646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.552630 4981 scope.go:122] "RemoveContainer" containerID="422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9" Nov 01 23:22:04.553865 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.553839 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:22:04.553935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.553913 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:22:04.553975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:04.553945 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:22:05.161631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.161572 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:05.441870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.441842 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:22:05.536065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.536035 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:05.559340 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.559307 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" event={"ID":"94f0ab30-8028-4576-a5b2-e82c42148ad8","Type":"ContainerStarted","Data":"703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc"} Nov 01 23:22:05.688239 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.688202 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:22:05.688650 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.688626 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" start-of-body= Nov 01 23:22:05.688855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.688831 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" Nov 01 23:22:05.865727 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:05.865641 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:06.118965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.118877 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:06.216634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.216599 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:06.317424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.317391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:06.691600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.683944 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" start-of-body= Nov 01 23:22:06.691600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.684012 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" Nov 01 23:22:06.750362 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.750309 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:06.962752 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.962670 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224"] Nov 01 23:22:06.966511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:06.966485 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.004913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.000473 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-45dd82\"/\"openshift-service-ca.crt\"" Nov 01 23:22:07.004913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.000667 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-45dd82\"/\"simulator-single-main-45dd82-kserve-self-signed-certs\"" Nov 01 23:22:07.004913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.000749 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-45dd82\"/\"default-dockercfg-ffl89\"" Nov 01 23:22:07.004913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.000842 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-45dd82\"/\"kube-root-ca.crt\"" Nov 01 23:22:07.010082 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.010057 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224"] Nov 01 23:22:07.012998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.012973 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:07.141550 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.141498 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-dshm\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.141764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.141675 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef70207-8cda-4362-a7a1-02df6addaa66-tls-certs\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.142005 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.141943 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-model-cache\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.142168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.142068 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4sm\" (UniqueName: \"kubernetes.io/projected/9ef70207-8cda-4362-a7a1-02df6addaa66-kube-api-access-jj4sm\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.142168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.142139 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-home\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.142316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.142170 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-tmp-dir\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.243608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.243510 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-model-cache\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.243608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.243549 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4sm\" (UniqueName: \"kubernetes.io/projected/9ef70207-8cda-4362-a7a1-02df6addaa66-kube-api-access-jj4sm\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.243608 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.243569 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-home\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.243948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.243619 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-tmp-dir\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.243948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.243656 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-dshm\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.243948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.243700 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef70207-8cda-4362-a7a1-02df6addaa66-tls-certs\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.244112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.244013 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-model-cache\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.244112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.244034 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-tmp-dir\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.244112 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.244069 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-home\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.246029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.246002 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-dshm\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.246290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.246275 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef70207-8cda-4362-a7a1-02df6addaa66-tls-certs\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.251738 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.251719 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4sm\" (UniqueName: \"kubernetes.io/projected/9ef70207-8cda-4362-a7a1-02df6addaa66-kube-api-access-jj4sm\") pod \"simulator-single-main-45dd82-kserve-6b8499bd55-m6224\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.311029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.310995 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:07.442947 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.442883 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:07.592672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.588063 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224"] Nov 01 23:22:07.606450 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:22:07.606415 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ef70207_8cda_4362_a7a1_02df6addaa66.slice/crio-902703513fc836dcd00393205625b67f5c67461b316ce5fa39f67a96f4bc74a2 WatchSource:0}: Error finding container 902703513fc836dcd00393205625b67f5c67461b316ce5fa39f67a96f4bc74a2: Status 404 returned error can't find the container with id 902703513fc836dcd00393205625b67f5c67461b316ce5fa39f67a96f4bc74a2 Nov 01 23:22:07.692175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.692017 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" start-of-body= Nov 01 23:22:07.692175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:07.692066 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" Nov 01 23:22:08.261743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:08.261712 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:08.599864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:08.599758 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" event={"ID":"9ef70207-8cda-4362-a7a1-02df6addaa66","Type":"ContainerStarted","Data":"9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd"} Nov 01 23:22:08.599864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:08.599812 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" event={"ID":"9ef70207-8cda-4362-a7a1-02df6addaa66","Type":"ContainerStarted","Data":"902703513fc836dcd00393205625b67f5c67461b316ce5fa39f67a96f4bc74a2"} Nov 01 23:22:08.690599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:08.682230 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" start-of-body= Nov 01 23:22:08.690599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:08.682305 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" Nov 01 23:22:09.299130 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:09.299081 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:09.311846 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:09.311818 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:09.675932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:09.675898 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" start-of-body= Nov 01 23:22:09.676202 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:09.676173 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.147:15021/healthz/ready\": dial tcp 10.134.1.147:15021: connect: connection refused" Nov 01 23:22:09.972020 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:09.971920 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:10.752016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:10.751989 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:11.037422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:11.037337 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:22:11.037715 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:11.037690 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:22:11.039641 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:11.039624 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:22:11.057290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:11.057079 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" podStartSLOduration=5.057062127 podStartE2EDuration="5.057062127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:22:08.647913551 +0000 UTC m=+3494.075791482" watchObservedRunningTime="2026-11-01 23:22:11.057062127 +0000 UTC m=+3496.484940059" Nov 01 23:22:11.235887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:11.235853 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:11.456512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:11.456483 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:12.236565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:12.236535 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:12.851645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:12.851614 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:13.528615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:13.528570 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:14.262422 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:14.262393 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:14.524046 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:22:14.523969 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:22:14.951569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:14.951535 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:15.634739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:15.634707 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:16.030764 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:16.030726 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:16.261769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:16.261739 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:16.460754 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:16.460722 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:16.865365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:16.865284 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:17.311524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:17.311489 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:17.325054 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:17.325027 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:17.517213 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:17.517167 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:17.656703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:17.656617 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:22:18.174551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:18.174514 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:18.863898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:18.863867 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:19.476099 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:19.476057 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:20.223996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:20.223969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:20.858307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:20.858277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:21.463390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:21.463356 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:22.269052 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:22.269021 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:22.958562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:22.958528 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:23.460337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:23.460304 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:23.551749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:23.551717 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:23.788181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:23.788104 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:24.466174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:24.466141 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:25.186664 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:25.186627 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:25.964177 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:25.964143 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:26.769462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:26.769423 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:27.458597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:27.458541 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:28.073731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:28.073696 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:28.782984 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:28.782955 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:29.466493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:29.466457 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:30.142514 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:30.142479 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:30.783221 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:30.783191 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:31.446427 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:31.446394 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:32.162158 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:32.162125 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:32.862850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:32.862816 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:33.574757 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:33.574724 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:34.077255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:34.077225 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:34.756488 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:34.756456 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:35.455336 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:35.455305 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:36.020461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:36.020429 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt"] Nov 01 23:22:36.022320 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:36.022296 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rs9qt"] Nov 01 23:22:36.053839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:36.053803 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:36.659939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:36.659904 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:36.733251 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:36.733218 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c575fa69-472e-41bf-b046-6d3b913025a1" path="/var/lib/kubelet/pods/c575fa69-472e-41bf-b046-6d3b913025a1/volumes" Nov 01 23:22:37.362693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:37.362662 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:38.156952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:38.156900 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:38.979646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:38.979615 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:40.138007 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:40.137978 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:40.368089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:40.368059 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:41.047315 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:41.047270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:41.670197 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:41.670169 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:42.253828 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:42.253797 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:42.933982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:42.933949 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:43.573339 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:43.573312 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:44.166641 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:44.166609 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:44.523206 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:22:44.523168 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:22:44.845351 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:44.845280 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:45.458061 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:45.458033 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:46.023779 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:46.023745 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h"] Nov 01 23:22:46.027863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:46.027836 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-jtj2h"] Nov 01 23:22:46.068464 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:46.068434 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:46.732275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:46.732235 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e559ead6-5242-4170-9871-7a45987b3eb3" path="/var/lib/kubelet/pods/e559ead6-5242-4170-9871-7a45987b3eb3/volumes" Nov 01 23:22:46.759420 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:46.759394 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:47.449525 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:47.449491 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:48.171528 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:48.171496 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:48.271597 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:48.271537 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:49.157502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:49.157467 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:49.768056 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:49.768025 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:50.435074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:50.435036 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:51.163970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:51.163891 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:51.758105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:51.758061 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:52.516386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:52.516356 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:53.162163 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:53.162130 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:53.163622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:53.163602 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224"] Nov 01 23:22:53.258669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:53.258629 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:53.780735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:53.780704 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:54.444921 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:54.444891 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:55.181174 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:55.181145 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:55.857470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:55.857429 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:56.467830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:56.467795 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:57.140630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:57.140567 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:57.750565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:57.750535 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:58.422977 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:58.422944 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:58.564482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:58.564448 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:58.660200 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:58.660173 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:59.115020 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.114988 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:59.587996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.587963 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h"] Nov 01 23:22:59.588210 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.588183 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" podUID="bc23ea8f-0587-4217-b70b-565c6c7716b4" containerName="maas-api" containerID="cri-o://a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc" gracePeriod=30 Nov 01 23:22:59.629927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.629899 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:59.749981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.749952 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:59.850232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.850168 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:22:59.853148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.853126 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:22:59.869611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.869557 4981 generic.go:356] "Generic (PLEG): container finished" podID="bc23ea8f-0587-4217-b70b-565c6c7716b4" containerID="a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc" exitCode=0 Nov 01 23:22:59.869737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.869617 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" Nov 01 23:22:59.869737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.869646 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" event={"ID":"bc23ea8f-0587-4217-b70b-565c6c7716b4","Type":"ContainerDied","Data":"a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc"} Nov 01 23:22:59.869737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.869700 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h" event={"ID":"bc23ea8f-0587-4217-b70b-565c6c7716b4","Type":"ContainerDied","Data":"bd4b56512efed8173bb96b915677c0c224e6050fc039af508f1780641f0bb27d"} Nov 01 23:22:59.869737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.869723 4981 scope.go:122] "RemoveContainer" containerID="a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc" Nov 01 23:22:59.881997 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.881976 4981 scope.go:122] "RemoveContainer" containerID="a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc" Nov 01 23:22:59.882334 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:22:59.882307 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc\": container with ID starting with a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc not found: ID does not exist" containerID="a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc" Nov 01 23:22:59.882405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:22:59.882350 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc"} err="failed to get container status \"a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc\": rpc error: code = NotFound desc = could not find container \"a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc\": container with ID starting with a9934e6377b18a558b451622ffb9c3bb68c671d6b8111093edb8fac207637cdc not found: ID does not exist" Nov 01 23:23:00.003810 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.003774 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc23ea8f-0587-4217-b70b-565c6c7716b4-kube-api-access-4bf5t\" (UniqueName: \"kubernetes.io/projected/bc23ea8f-0587-4217-b70b-565c6c7716b4-kube-api-access-4bf5t\") pod \"bc23ea8f-0587-4217-b70b-565c6c7716b4\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " Nov 01 23:23:00.004010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.003903 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc23ea8f-0587-4217-b70b-565c6c7716b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc23ea8f-0587-4217-b70b-565c6c7716b4-openshift-service-ca\") pod \"bc23ea8f-0587-4217-b70b-565c6c7716b4\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " Nov 01 23:23:00.004010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.003940 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-maas-api-tls\") pod \"bc23ea8f-0587-4217-b70b-565c6c7716b4\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " Nov 01 23:23:00.004010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.003974 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-metrics-certs\") pod \"bc23ea8f-0587-4217-b70b-565c6c7716b4\" (UID: \"bc23ea8f-0587-4217-b70b-565c6c7716b4\") " Nov 01 23:23:00.004349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.004314 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc23ea8f-0587-4217-b70b-565c6c7716b4-openshift-service-ca" pod "bc23ea8f-0587-4217-b70b-565c6c7716b4" (UID: "bc23ea8f-0587-4217-b70b-565c6c7716b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:23:00.006135 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.006108 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-maas-api-tls" pod "bc23ea8f-0587-4217-b70b-565c6c7716b4" (UID: "bc23ea8f-0587-4217-b70b-565c6c7716b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:23:00.006252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.006139 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-metrics-certs" pod "bc23ea8f-0587-4217-b70b-565c6c7716b4" (UID: "bc23ea8f-0587-4217-b70b-565c6c7716b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:23:00.006252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.006176 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc23ea8f-0587-4217-b70b-565c6c7716b4-kube-api-access-4bf5t" pod "bc23ea8f-0587-4217-b70b-565c6c7716b4" (UID: "bc23ea8f-0587-4217-b70b-565c6c7716b4"). InnerVolumeSpecName "kube-api-access-4bf5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:23:00.105481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.105399 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc23ea8f-0587-4217-b70b-565c6c7716b4-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:00.105481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.105427 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:00.105481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.105436 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc23ea8f-0587-4217-b70b-565c6c7716b4-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:00.105481 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.105444 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bf5t\" (UniqueName: \"kubernetes.io/projected/bc23ea8f-0587-4217-b70b-565c6c7716b4-kube-api-access-4bf5t\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:00.192028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.192000 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h"] Nov 01 23:23:00.195817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.195793 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-45dd82-859c9d7d75-kjv6h"] Nov 01 23:23:00.308965 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.308934 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:00.733002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:00.732970 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc23ea8f-0587-4217-b70b-565c6c7716b4" path="/var/lib/kubelet/pods/bc23ea8f-0587-4217-b70b-565c6c7716b4/volumes" Nov 01 23:23:04.341879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:04.341846 4981 scope.go:122] "RemoveContainer" containerID="9a70768742648ace79799bba88e6bad99e3abcc3ff2f8c12c07eb612ed7e55e4" Nov 01 23:23:04.350703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:04.350653 4981 scope.go:122] "RemoveContainer" containerID="bdc19c387eeebf3635e23eaf8cabcffe285ecc435ec2eeddd09fd4ae214ccc7c" Nov 01 23:23:04.358634 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:04.358605 4981 scope.go:122] "RemoveContainer" containerID="24f0c5886a37d5aa83881ee0734e8c6230be19eac39ac75469f96438286628f8" Nov 01 23:23:06.549926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:06.549888 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:06.650627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:06.650592 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:06.844262 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:06.844166 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p"] Nov 01 23:23:06.844423 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:06.844381 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" containerID="cri-o://703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc" gracePeriod=30 Nov 01 23:23:06.888167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:06.888110 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:08.184344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.184302 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" podUID="9ef70207-8cda-4362-a7a1-02df6addaa66" containerName="main" containerID="cri-o://9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd" gracePeriod=45 Nov 01 23:23:08.437814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.437747 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:23:08.593110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593077 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-model-cache\") pod \"9ef70207-8cda-4362-a7a1-02df6addaa66\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " Nov 01 23:23:08.593110 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593121 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef70207-8cda-4362-a7a1-02df6addaa66-kube-api-access-jj4sm\" (UniqueName: \"kubernetes.io/projected/9ef70207-8cda-4362-a7a1-02df6addaa66-kube-api-access-jj4sm\") pod \"9ef70207-8cda-4362-a7a1-02df6addaa66\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " Nov 01 23:23:08.593388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593153 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-tmp-dir\") pod \"9ef70207-8cda-4362-a7a1-02df6addaa66\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " Nov 01 23:23:08.593388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593178 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-home\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-home\") pod \"9ef70207-8cda-4362-a7a1-02df6addaa66\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " Nov 01 23:23:08.593388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593330 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-dshm\") pod \"9ef70207-8cda-4362-a7a1-02df6addaa66\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " Nov 01 23:23:08.593388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593371 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef70207-8cda-4362-a7a1-02df6addaa66-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef70207-8cda-4362-a7a1-02df6addaa66-tls-certs\") pod \"9ef70207-8cda-4362-a7a1-02df6addaa66\" (UID: \"9ef70207-8cda-4362-a7a1-02df6addaa66\") " Nov 01 23:23:08.593659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593408 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-tmp-dir" pod "9ef70207-8cda-4362-a7a1-02df6addaa66" (UID: "9ef70207-8cda-4362-a7a1-02df6addaa66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:08.593659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593422 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-model-cache" pod "9ef70207-8cda-4362-a7a1-02df6addaa66" (UID: "9ef70207-8cda-4362-a7a1-02df6addaa66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:08.593659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593447 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-home" pod "9ef70207-8cda-4362-a7a1-02df6addaa66" (UID: "9ef70207-8cda-4362-a7a1-02df6addaa66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:08.593813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593712 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:08.593813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593732 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:08.593813 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.593748 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:08.595479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.595454 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-dshm" pod "9ef70207-8cda-4362-a7a1-02df6addaa66" (UID: "9ef70207-8cda-4362-a7a1-02df6addaa66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:08.595610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.595504 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef70207-8cda-4362-a7a1-02df6addaa66-tls-certs" pod "9ef70207-8cda-4362-a7a1-02df6addaa66" (UID: "9ef70207-8cda-4362-a7a1-02df6addaa66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:23:08.595610 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.595536 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef70207-8cda-4362-a7a1-02df6addaa66-kube-api-access-jj4sm" pod "9ef70207-8cda-4362-a7a1-02df6addaa66" (UID: "9ef70207-8cda-4362-a7a1-02df6addaa66"). InnerVolumeSpecName "kube-api-access-jj4sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:23:08.695048 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.694969 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ef70207-8cda-4362-a7a1-02df6addaa66-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:08.695048 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.694997 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ef70207-8cda-4362-a7a1-02df6addaa66-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:08.695048 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.695007 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj4sm\" (UniqueName: \"kubernetes.io/projected/9ef70207-8cda-4362-a7a1-02df6addaa66-kube-api-access-jj4sm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:08.926297 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.926255 4981 generic.go:356] "Generic (PLEG): container finished" podID="9ef70207-8cda-4362-a7a1-02df6addaa66" containerID="9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd" exitCode=0 Nov 01 23:23:08.926493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.926298 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" event={"ID":"9ef70207-8cda-4362-a7a1-02df6addaa66","Type":"ContainerDied","Data":"9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd"} Nov 01 23:23:08.926493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.926325 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" Nov 01 23:23:08.926493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.926349 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224" event={"ID":"9ef70207-8cda-4362-a7a1-02df6addaa66","Type":"ContainerDied","Data":"902703513fc836dcd00393205625b67f5c67461b316ce5fa39f67a96f4bc74a2"} Nov 01 23:23:08.926493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.926362 4981 scope.go:122] "RemoveContainer" containerID="9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd" Nov 01 23:23:08.935189 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.935170 4981 scope.go:122] "RemoveContainer" containerID="9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd" Nov 01 23:23:08.935475 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:23:08.935457 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd\": container with ID starting with 9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd not found: ID does not exist" containerID="9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd" Nov 01 23:23:08.935527 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.935486 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd"} err="failed to get container status \"9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd\": rpc error: code = NotFound desc = could not find container \"9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd\": container with ID starting with 9711fd9230bf401daaaaf2731a16b06289133ce1f69eb27ccf51e1e99e5ed0bd not found: ID does not exist" Nov 01 23:23:08.944047 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.944015 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224"] Nov 01 23:23:08.947854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:08.947797 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-45dd82/simulator-single-main-45dd82-kserve-6b8499bd55-m6224"] Nov 01 23:23:10.732512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.732479 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef70207-8cda-4362-a7a1-02df6addaa66" path="/var/lib/kubelet/pods/9ef70207-8cda-4362-a7a1-02df6addaa66/volumes" Nov 01 23:23:10.985395 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.985303 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk"] Nov 01 23:23:10.986092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.986067 4981 cpu_manager.go:414] "containerMap: removing container" podUID="9ef70207-8cda-4362-a7a1-02df6addaa66" containerName="main" Nov 01 23:23:10.986166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.986097 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef70207-8cda-4362-a7a1-02df6addaa66" containerName="main" Nov 01 23:23:10.986166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.986128 4981 cpu_manager.go:414] "containerMap: removing container" podUID="bc23ea8f-0587-4217-b70b-565c6c7716b4" containerName="maas-api" Nov 01 23:23:10.986166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.986137 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc23ea8f-0587-4217-b70b-565c6c7716b4" containerName="maas-api" Nov 01 23:23:10.986342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.986323 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef70207-8cda-4362-a7a1-02df6addaa66" containerName="main" Nov 01 23:23:10.986381 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.986349 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc23ea8f-0587-4217-b70b-565c6c7716b4" containerName="maas-api" Nov 01 23:23:10.991621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.991599 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:10.996674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.996649 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk"] Nov 01 23:23:10.996962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:10.996942 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d660b8b1-kserve-self-signed-certs\"" Nov 01 23:23:11.014144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.014099 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tls-certs\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.014144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.014135 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tmp-dir\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.014401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.014164 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2vkh\" (UniqueName: \"kubernetes.io/projected/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-kube-api-access-m2vkh\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.014401 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.014359 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-home\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.014532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.014397 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-model-cache\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.014532 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.014482 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-dshm\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.040688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.040647 4981 patch_prober.go:29] interesting pod/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:23:11.040887 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.040708 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:23:11.115830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.115790 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-home\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.115830 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.115841 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-model-cache\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.116111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.115889 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-dshm\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.116111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.115938 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tls-certs\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.116111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.115961 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tmp-dir\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.116111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.116002 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2vkh\" (UniqueName: \"kubernetes.io/projected/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-kube-api-access-m2vkh\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.116346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.116308 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-home\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.116645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.116614 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tmp-dir\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.117037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.116308 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-model-cache\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.119558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.119533 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-dshm\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.119970 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.119921 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tls-certs\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.127365 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.127333 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2vkh\" (UniqueName: \"kubernetes.io/projected/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-kube-api-access-m2vkh\") pod \"e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.145416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.145388 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:11.308470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.308386 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:11.583840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.583812 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk"] Nov 01 23:23:11.585276 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:23:11.585250 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa09d70f_c98e_4f6c_970a_2bcae3c3aa5d.slice/crio-e29cecb366195487fa719a9d396924f20c327d23eb057fcacef01fbaa047322f WatchSource:0}: Error finding container e29cecb366195487fa719a9d396924f20c327d23eb057fcacef01fbaa047322f: Status 404 returned error can't find the container with id e29cecb366195487fa719a9d396924f20c327d23eb057fcacef01fbaa047322f Nov 01 23:23:11.943982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.943947 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" event={"ID":"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d","Type":"ContainerStarted","Data":"312e6528639f5b0216edfc9510b2f5728b4fe52491a3ace49f4a82b24b5f28b9"} Nov 01 23:23:11.943982 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.943986 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" event={"ID":"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d","Type":"ContainerStarted","Data":"e29cecb366195487fa719a9d396924f20c327d23eb057fcacef01fbaa047322f"} Nov 01 23:23:11.960991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:11.960948 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" podStartSLOduration=1.9609365890000001 podStartE2EDuration="1.960936589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:23:11.960213343 +0000 UTC m=+3557.388091287" watchObservedRunningTime="2026-11-01 23:23:11.960936589 +0000 UTC m=+3557.388814520" Nov 01 23:23:12.090594 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.090551 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:23:12.127218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127021 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-token\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-token\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127081 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-socket\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127218 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127129 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-credential-socket\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127225 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-data\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127257 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-certs\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127419 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-podinfo\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127484 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-kube-api-access-q99f9\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-kube-api-access-q99f9\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127533 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f0ab30-8028-4576-a5b2-e82c42148ad8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f0ab30-8028-4576-a5b2-e82c42148ad8-istiod-ca-cert\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.127638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127512 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-socket" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:12.127638 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.127586 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-envoy\") pod \"94f0ab30-8028-4576-a5b2-e82c42148ad8\" (UID: \"94f0ab30-8028-4576-a5b2-e82c42148ad8\") " Nov 01 23:23:12.128373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128045 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-data" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:12.128373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128071 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-credential-socket" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:12.128373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128131 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-certs" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:12.128373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128348 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f0ab30-8028-4576-a5b2-e82c42148ad8-istiod-ca-cert" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:23:12.128778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128544 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f0ab30-8028-4576-a5b2-e82c42148ad8-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.128778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128562 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.128778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128606 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.128778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128621 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.128778 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.128634 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.130154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.130122 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-envoy" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:23:12.130615 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.130563 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-podinfo" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:23:12.130824 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.130804 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-token" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:23:12.130919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.130899 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-kube-api-access-q99f9" pod "94f0ab30-8028-4576-a5b2-e82c42148ad8" (UID: "94f0ab30-8028-4576-a5b2-e82c42148ad8"). InnerVolumeSpecName "kube-api-access-q99f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:23:12.230171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.230135 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.230171 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.230169 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99f9\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-kube-api-access-q99f9\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.230386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.230182 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.230386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.230191 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f0ab30-8028-4576-a5b2-e82c42148ad8-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:12.308543 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.308509 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:12.951131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.951098 4981 generic.go:356] "Generic (PLEG): container finished" podID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerID="703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc" exitCode=0 Nov 01 23:23:12.951645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.951143 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" Nov 01 23:23:12.951645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.951183 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" event={"ID":"94f0ab30-8028-4576-a5b2-e82c42148ad8","Type":"ContainerDied","Data":"703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc"} Nov 01 23:23:12.951645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.951244 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p" event={"ID":"94f0ab30-8028-4576-a5b2-e82c42148ad8","Type":"ContainerDied","Data":"ecc289ec99a50b020abc680af9d53eab52b0a795275d1513699c550f93f537b0"} Nov 01 23:23:12.951645 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.951276 4981 scope.go:122] "RemoveContainer" containerID="703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc" Nov 01 23:23:12.960393 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.960375 4981 scope.go:122] "RemoveContainer" containerID="422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9" Nov 01 23:23:12.968438 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.968420 4981 scope.go:122] "RemoveContainer" containerID="703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc" Nov 01 23:23:12.968779 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:23:12.968762 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc\": container with ID starting with 703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc not found: ID does not exist" containerID="703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc" Nov 01 23:23:12.968839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.968787 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc"} err="failed to get container status \"703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc\": rpc error: code = NotFound desc = could not find container \"703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc\": container with ID starting with 703ecf7f7f94d78c718eb04071994233cc1a87a4c8e766ca49f55134111a1acc not found: ID does not exist" Nov 01 23:23:12.968839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.968804 4981 scope.go:122] "RemoveContainer" containerID="422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9" Nov 01 23:23:12.969027 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:23:12.969011 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9\": container with ID starting with 422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9 not found: ID does not exist" containerID="422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9" Nov 01 23:23:12.969072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.969031 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9"} err="failed to get container status \"422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9\": rpc error: code = NotFound desc = could not find container \"422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9\": container with ID starting with 422f43e2018fe4dff98064b035496280fae8b52ed6bae742287b230dded793e9 not found: ID does not exist" Nov 01 23:23:12.973699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.973678 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p"] Nov 01 23:23:12.978624 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:12.978604 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-45dd82-openshift-default-5c4bd49b7c6gx9p"] Nov 01 23:23:14.566726 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:23:14.566689 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:23:14.747512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:14.747475 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" path="/var/lib/kubelet/pods/94f0ab30-8028-4576-a5b2-e82c42148ad8/volumes" Nov 01 23:23:15.118748 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:15.118701 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6tll8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:15.119077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:15.119045 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6tll8" podUID="3684469b-976a-4854-9697-a4d97edc5b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:15.450598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:15.449785 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lt55h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:15.450598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:15.449882 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lt55h" podUID="68b388b0-9081-4e9f-9bc4-c0f4b91397fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:16.083344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:16.083309 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:21.309268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:21.309228 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:21.321681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:21.321653 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:22.009855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:22.009817 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:23:27.608699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.608653 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p"] Nov 01 23:23:27.609248 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.609076 4981 cpu_manager.go:414] "containerMap: removing container" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" Nov 01 23:23:27.609248 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.609090 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" Nov 01 23:23:27.609248 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.609215 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" Nov 01 23:23:27.612402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.612382 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.623096 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.623066 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p"] Nov 01 23:23:27.624619 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.624596 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d660b8b1-kserve-self-signed-certs\"" Nov 01 23:23:27.661775 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.661739 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:27.687990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.687962 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9p2\" (UniqueName: \"kubernetes.io/projected/4419e8cb-f3e9-4f01-8253-fa90515daa53-kube-api-access-gb9p2\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.688123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.688017 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-home\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.688123 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.688095 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-model-cache\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.688211 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.688142 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-dshm\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.688211 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.688163 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-tmp-dir\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.688211 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.688183 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4419e8cb-f3e9-4f01-8253-fa90515daa53-tls-certs\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789206 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-home\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789263 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-model-cache\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789323 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-dshm\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789347 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-tmp-dir\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789376 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4419e8cb-f3e9-4f01-8253-fa90515daa53-tls-certs\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789470 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789445 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9p2\" (UniqueName: \"kubernetes.io/projected/4419e8cb-f3e9-4f01-8253-fa90515daa53-kube-api-access-gb9p2\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789672 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-home\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789706 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-model-cache\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.789842 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.789725 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-tmp-dir\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.791688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.791660 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-dshm\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.791918 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.791899 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4419e8cb-f3e9-4f01-8253-fa90515daa53-tls-certs\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.796863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.796841 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9p2\" (UniqueName: \"kubernetes.io/projected/4419e8cb-f3e9-4f01-8253-fa90515daa53-kube-api-access-gb9p2\") pod \"e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:27.926870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:27.926847 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:28.200523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:28.200498 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p"] Nov 01 23:23:28.202334 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:23:28.202305 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4419e8cb_f3e9_4f01_8253_fa90515daa53.slice/crio-975503d44777f8c594063cbd8fac9b8805cfbfbeb372c37834f28ac01dd7f7b1 WatchSource:0}: Error finding container 975503d44777f8c594063cbd8fac9b8805cfbfbeb372c37834f28ac01dd7f7b1: Status 404 returned error can't find the container with id 975503d44777f8c594063cbd8fac9b8805cfbfbeb372c37834f28ac01dd7f7b1 Nov 01 23:23:29.040899 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:29.040862 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" event={"ID":"4419e8cb-f3e9-4f01-8253-fa90515daa53","Type":"ContainerStarted","Data":"50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681"} Nov 01 23:23:29.041331 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:29.040908 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" event={"ID":"4419e8cb-f3e9-4f01-8253-fa90515daa53","Type":"ContainerStarted","Data":"975503d44777f8c594063cbd8fac9b8805cfbfbeb372c37834f28ac01dd7f7b1"} Nov 01 23:23:29.057872 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:29.057808 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" podStartSLOduration=2.057792144 podStartE2EDuration="2.057792144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:23:29.057281846 +0000 UTC m=+3574.485159778" watchObservedRunningTime="2026-11-01 23:23:29.057792144 +0000 UTC m=+3574.485670075" Nov 01 23:23:29.403491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:29.403409 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:29.927595 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:29.927537 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:30.986611 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:30.986269 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:32.279777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:32.279723 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:32.280463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:32.280424 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:36.775896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:36.775861 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:37.927217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:37.927169 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:37.940380 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:37.940352 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:38.102719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:38.102686 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:23:44.307471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.307439 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv"] Nov 01 23:23:44.307903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.307889 4981 cpu_manager.go:414] "containerMap: removing container" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" Nov 01 23:23:44.307903 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.307903 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" Nov 01 23:23:44.307992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.307973 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f0ab30-8028-4576-a5b2-e82c42148ad8" containerName="istio-proxy" Nov 01 23:23:44.311247 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.311231 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.316898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.316870 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:23:44.317045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.316932 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:23:44.317045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.316940 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:23:44.317168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.317095 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:23:44.318446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.318423 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv"] Nov 01 23:23:44.377782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.377749 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.377782 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.377786 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kvl\" (UniqueName: \"kubernetes.io/projected/82f93db3-fe39-413f-8fa3-3c1746108d62-kube-api-access-v2kvl\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.378024 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.377807 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.479146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.479100 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.479146 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.479149 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kvl\" (UniqueName: \"kubernetes.io/projected/82f93db3-fe39-413f-8fa3-3c1746108d62-kube-api-access-v2kvl\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.479504 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.479168 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.479838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.479815 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.481551 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.481529 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.486696 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.486677 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kvl\" (UniqueName: \"kubernetes.io/projected/82f93db3-fe39-413f-8fa3-3c1746108d62-kube-api-access-v2kvl\") pod \"maas-api-revoke-sub-452b6d7b-6zgsv\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.524494 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:23:44.524456 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:23:44.626996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.626904 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:44.901699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:44.901670 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv"] Nov 01 23:23:44.904410 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:23:44.904376 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f93db3_fe39_413f_8fa3_3c1746108d62.slice/crio-273bfb4674c78015f6c1fb8494f8e129eb3d2b5126068b98ecf3e872cc2b5b55 WatchSource:0}: Error finding container 273bfb4674c78015f6c1fb8494f8e129eb3d2b5126068b98ecf3e872cc2b5b55: Status 404 returned error can't find the container with id 273bfb4674c78015f6c1fb8494f8e129eb3d2b5126068b98ecf3e872cc2b5b55 Nov 01 23:23:45.127009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:45.126968 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" event={"ID":"82f93db3-fe39-413f-8fa3-3c1746108d62","Type":"ContainerStarted","Data":"273bfb4674c78015f6c1fb8494f8e129eb3d2b5126068b98ecf3e872cc2b5b55"} Nov 01 23:23:46.135733 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:46.135635 4981 generic.go:356] "Generic (PLEG): container finished" podID="82f93db3-fe39-413f-8fa3-3c1746108d62" containerID="751bcbfc5ad81464ec03bd9a8d3ba5e8a455245f9362958f5c525e7029f6e96d" exitCode=0 Nov 01 23:23:46.136137 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:46.135745 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" event={"ID":"82f93db3-fe39-413f-8fa3-3c1746108d62","Type":"ContainerDied","Data":"751bcbfc5ad81464ec03bd9a8d3ba5e8a455245f9362958f5c525e7029f6e96d"} Nov 01 23:23:47.291805 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.291738 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:23:47.292456 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.292412 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:23:47.764765 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.764733 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:47.847598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.840443 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f93db3-fe39-413f-8fa3-3c1746108d62-kube-api-access-v2kvl\" (UniqueName: \"kubernetes.io/projected/82f93db3-fe39-413f-8fa3-3c1746108d62-kube-api-access-v2kvl\") pod \"82f93db3-fe39-413f-8fa3-3c1746108d62\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " Nov 01 23:23:47.847966 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.847945 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-openshift-ca\") pod \"82f93db3-fe39-413f-8fa3-3c1746108d62\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " Nov 01 23:23:47.848193 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.848177 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-xks-ca\") pod \"82f93db3-fe39-413f-8fa3-3c1746108d62\" (UID: \"82f93db3-fe39-413f-8fa3-3c1746108d62\") " Nov 01 23:23:47.848993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.848009 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f93db3-fe39-413f-8fa3-3c1746108d62-kube-api-access-v2kvl" pod "82f93db3-fe39-413f-8fa3-3c1746108d62" (UID: "82f93db3-fe39-413f-8fa3-3c1746108d62"). InnerVolumeSpecName "kube-api-access-v2kvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:23:47.849145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.848327 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-openshift-ca" pod "82f93db3-fe39-413f-8fa3-3c1746108d62" (UID: "82f93db3-fe39-413f-8fa3-3c1746108d62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:23:47.869996 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.869952 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-xks-ca" pod "82f93db3-fe39-413f-8fa3-3c1746108d62" (UID: "82f93db3-fe39-413f-8fa3-3c1746108d62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:23:47.950066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.950035 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:47.950296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.950277 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2kvl\" (UniqueName: \"kubernetes.io/projected/82f93db3-fe39-413f-8fa3-3c1746108d62-kube-api-access-v2kvl\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:47.950412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:47.950398 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f93db3-fe39-413f-8fa3-3c1746108d62-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:23:48.167568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:48.167472 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" Nov 01 23:23:48.167568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:48.167493 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv" event={"ID":"82f93db3-fe39-413f-8fa3-3c1746108d62","Type":"ContainerDied","Data":"273bfb4674c78015f6c1fb8494f8e129eb3d2b5126068b98ecf3e872cc2b5b55"} Nov 01 23:23:48.167568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:48.167566 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="273bfb4674c78015f6c1fb8494f8e129eb3d2b5126068b98ecf3e872cc2b5b55" Nov 01 23:23:55.118090 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:55.118057 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk"] Nov 01 23:23:55.166888 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:55.166851 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:23:56.735378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:56.735344 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p"] Nov 01 23:23:56.785222 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:23:56.785186 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:00.415510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.415477 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq"] Nov 01 23:24:00.416151 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.416130 4981 cpu_manager.go:414] "containerMap: removing container" podUID="82f93db3-fe39-413f-8fa3-3c1746108d62" containerName="revoke-keys" Nov 01 23:24:00.416246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.416153 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f93db3-fe39-413f-8fa3-3c1746108d62" containerName="revoke-keys" Nov 01 23:24:00.416332 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.416319 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f93db3-fe39-413f-8fa3-3c1746108d62" containerName="revoke-keys" Nov 01 23:24:00.419548 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.419525 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.425633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.425373 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:24:00.427461 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.426241 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:24:00.446284 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.446234 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq"] Nov 01 23:24:00.463405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.463369 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:24:00.463791 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.463768 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:24:00.586103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.581740 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkft\" (UniqueName: \"kubernetes.io/projected/86452723-09a0-4e7e-8f6c-21c2c93a3439-kube-api-access-nqkft\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.586103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.581908 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.586103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.581989 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.682759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.682701 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.682988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.682816 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.682988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.682910 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkft\" (UniqueName: \"kubernetes.io/projected/86452723-09a0-4e7e-8f6c-21c2c93a3439-kube-api-access-nqkft\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.683660 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.683625 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.685354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.685329 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.692048 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.692021 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkft\" (UniqueName: \"kubernetes.io/projected/86452723-09a0-4e7e-8f6c-21c2c93a3439-kube-api-access-nqkft\") pod \"maas-api-revoke-sub-3e31a506-t6xjq\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:00.747945 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:00.747890 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:01.030064 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:01.030035 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq"] Nov 01 23:24:01.032405 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:24:01.032363 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86452723_09a0_4e7e_8f6c_21c2c93a3439.slice/crio-80bd877cf19f0f7a548f8b7ed8596011f7c9c4d554afdc5d6896c816a6cfd9f8 WatchSource:0}: Error finding container 80bd877cf19f0f7a548f8b7ed8596011f7c9c4d554afdc5d6896c816a6cfd9f8: Status 404 returned error can't find the container with id 80bd877cf19f0f7a548f8b7ed8596011f7c9c4d554afdc5d6896c816a6cfd9f8 Nov 01 23:24:01.281167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:01.280996 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" event={"ID":"86452723-09a0-4e7e-8f6c-21c2c93a3439","Type":"ContainerStarted","Data":"80bd877cf19f0f7a548f8b7ed8596011f7c9c4d554afdc5d6896c816a6cfd9f8"} Nov 01 23:24:02.283533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:02.282037 4981 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-7v6z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 23:24:02.283533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:02.282130 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-7v6z5" podUID="4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 23:24:02.317600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:02.316795 4981 generic.go:356] "Generic (PLEG): container finished" podID="86452723-09a0-4e7e-8f6c-21c2c93a3439" containerID="592b4a6dfafb35ebd72094818c28b6c8ee9ea59535ab4bddbe1c7ee6ac134b73" exitCode=0 Nov 01 23:24:02.317600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:02.316925 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" event={"ID":"86452723-09a0-4e7e-8f6c-21c2c93a3439","Type":"ContainerDied","Data":"592b4a6dfafb35ebd72094818c28b6c8ee9ea59535ab4bddbe1c7ee6ac134b73"} Nov 01 23:24:03.464051 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.464024 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:03.556065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.556023 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-xks-ca\") pod \"86452723-09a0-4e7e-8f6c-21c2c93a3439\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " Nov 01 23:24:03.556065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.556072 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-openshift-ca\") pod \"86452723-09a0-4e7e-8f6c-21c2c93a3439\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " Nov 01 23:24:03.556373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.556168 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86452723-09a0-4e7e-8f6c-21c2c93a3439-kube-api-access-nqkft\" (UniqueName: \"kubernetes.io/projected/86452723-09a0-4e7e-8f6c-21c2c93a3439-kube-api-access-nqkft\") pod \"86452723-09a0-4e7e-8f6c-21c2c93a3439\" (UID: \"86452723-09a0-4e7e-8f6c-21c2c93a3439\") " Nov 01 23:24:03.556521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.556490 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-openshift-ca" pod "86452723-09a0-4e7e-8f6c-21c2c93a3439" (UID: "86452723-09a0-4e7e-8f6c-21c2c93a3439"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:24:03.558505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.558470 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-xks-ca" pod "86452723-09a0-4e7e-8f6c-21c2c93a3439" (UID: "86452723-09a0-4e7e-8f6c-21c2c93a3439"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:03.558505 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.558490 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86452723-09a0-4e7e-8f6c-21c2c93a3439-kube-api-access-nqkft" pod "86452723-09a0-4e7e-8f6c-21c2c93a3439" (UID: "86452723-09a0-4e7e-8f6c-21c2c93a3439"). InnerVolumeSpecName "kube-api-access-nqkft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:24:03.657311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.657224 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:03.657311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.657254 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86452723-09a0-4e7e-8f6c-21c2c93a3439-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:03.657311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:03.657266 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqkft\" (UniqueName: \"kubernetes.io/projected/86452723-09a0-4e7e-8f6c-21c2c93a3439-kube-api-access-nqkft\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:04.329094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:04.329058 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" Nov 01 23:24:04.329094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:04.329081 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq" event={"ID":"86452723-09a0-4e7e-8f6c-21c2c93a3439","Type":"ContainerDied","Data":"80bd877cf19f0f7a548f8b7ed8596011f7c9c4d554afdc5d6896c816a6cfd9f8"} Nov 01 23:24:04.329325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:04.329135 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bd877cf19f0f7a548f8b7ed8596011f7c9c4d554afdc5d6896c816a6cfd9f8" Nov 01 23:24:04.391704 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:04.391673 4981 scope.go:122] "RemoveContainer" containerID="f2a4baae13da0cf182c1c786157e4b49fb9aac42d5a87b65cbc8b53dd108e410" Nov 01 23:24:04.400503 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:04.400483 4981 scope.go:122] "RemoveContainer" containerID="274cad31625790620621a30b410f548d67380cad2dbed9cceacb155ee717c266" Nov 01 23:24:04.408424 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:04.408400 4981 scope.go:122] "RemoveContainer" containerID="56f8ad47d0ee2b2ad46339f13f18fefc88ba727d6e67d749b71406aa3ecd70f5" Nov 01 23:24:04.416071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:04.416048 4981 scope.go:122] "RemoveContainer" containerID="391c2fa24a86587f0c171f6f510d02617a11cec9a74756d3d1e764227ec98abe" Nov 01 23:24:10.139306 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.139200 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" podUID="fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" containerName="main" containerID="cri-o://312e6528639f5b0216edfc9510b2f5728b4fe52491a3ace49f4a82b24b5f28b9" gracePeriod=45 Nov 01 23:24:10.360355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.360324 4981 generic.go:356] "Generic (PLEG): container finished" podID="fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" containerID="312e6528639f5b0216edfc9510b2f5728b4fe52491a3ace49f4a82b24b5f28b9" exitCode=0 Nov 01 23:24:10.360533 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.360391 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" event={"ID":"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d","Type":"ContainerDied","Data":"312e6528639f5b0216edfc9510b2f5728b4fe52491a3ace49f4a82b24b5f28b9"} Nov 01 23:24:10.399029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.398971 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:24:10.423593 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423551 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tmp-dir\") pod \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " Nov 01 23:24:10.423769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423660 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-model-cache\") pod \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " Nov 01 23:24:10.423769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423708 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tls-certs\") pod \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " Nov 01 23:24:10.423769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423757 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-home\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-home\") pod \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " Nov 01 23:24:10.423992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423795 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-dshm\") pod \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " Nov 01 23:24:10.423992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423843 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tmp-dir" pod "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" (UID: "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:10.423992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423863 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-kube-api-access-m2vkh\" (UniqueName: \"kubernetes.io/projected/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-kube-api-access-m2vkh\") pod \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\" (UID: \"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d\") " Nov 01 23:24:10.423992 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.423952 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-model-cache" pod "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" (UID: "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:10.424559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.424069 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-home" pod "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" (UID: "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:10.424559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.424276 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:10.424559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.424296 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:10.424559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.424310 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:10.426140 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.426104 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-kube-api-access-m2vkh" pod "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" (UID: "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d"). InnerVolumeSpecName "kube-api-access-m2vkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:24:10.426386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.426356 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tls-certs" pod "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" (UID: "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:10.426517 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.426389 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-dshm" pod "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" (UID: "fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:10.525493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.525463 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:10.525493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.525493 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2vkh\" (UniqueName: \"kubernetes.io/projected/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-kube-api-access-m2vkh\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:10.525737 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:10.525504 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:11.366723 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:11.366694 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" Nov 01 23:24:11.367157 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:11.366689 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk" event={"ID":"fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d","Type":"ContainerDied","Data":"e29cecb366195487fa719a9d396924f20c327d23eb057fcacef01fbaa047322f"} Nov 01 23:24:11.367157 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:11.366825 4981 scope.go:122] "RemoveContainer" containerID="312e6528639f5b0216edfc9510b2f5728b4fe52491a3ace49f4a82b24b5f28b9" Nov 01 23:24:11.383288 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:11.383263 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk"] Nov 01 23:24:11.386631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:11.386608 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d660b8b1-kserve-65c6c9c54f-k2cbk"] Nov 01 23:24:11.763988 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:11.763811 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" podUID="4419e8cb-f3e9-4f01-8253-fa90515daa53" containerName="main" containerID="cri-o://50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681" gracePeriod=45 Nov 01 23:24:12.296807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.296774 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:24:12.355653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.354901 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-model-cache\") pod \"4419e8cb-f3e9-4f01-8253-fa90515daa53\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " Nov 01 23:24:12.355653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.354960 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-tmp-dir\") pod \"4419e8cb-f3e9-4f01-8253-fa90515daa53\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " Nov 01 23:24:12.355653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.355006 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-dshm\") pod \"4419e8cb-f3e9-4f01-8253-fa90515daa53\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " Nov 01 23:24:12.355653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.355048 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4419e8cb-f3e9-4f01-8253-fa90515daa53-tls-certs\" (UniqueName: \"kubernetes.io/secret/4419e8cb-f3e9-4f01-8253-fa90515daa53-tls-certs\") pod \"4419e8cb-f3e9-4f01-8253-fa90515daa53\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " Nov 01 23:24:12.355653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.355109 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4419e8cb-f3e9-4f01-8253-fa90515daa53-kube-api-access-gb9p2\" (UniqueName: \"kubernetes.io/projected/4419e8cb-f3e9-4f01-8253-fa90515daa53-kube-api-access-gb9p2\") pod \"4419e8cb-f3e9-4f01-8253-fa90515daa53\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " Nov 01 23:24:12.355653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.355202 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-home\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-home\") pod \"4419e8cb-f3e9-4f01-8253-fa90515daa53\" (UID: \"4419e8cb-f3e9-4f01-8253-fa90515daa53\") " Nov 01 23:24:12.356142 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.355818 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-home" pod "4419e8cb-f3e9-4f01-8253-fa90515daa53" (UID: "4419e8cb-f3e9-4f01-8253-fa90515daa53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:12.357643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.357566 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-model-cache" pod "4419e8cb-f3e9-4f01-8253-fa90515daa53" (UID: "4419e8cb-f3e9-4f01-8253-fa90515daa53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:12.357643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.357600 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-tmp-dir" pod "4419e8cb-f3e9-4f01-8253-fa90515daa53" (UID: "4419e8cb-f3e9-4f01-8253-fa90515daa53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:12.359958 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.359935 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4419e8cb-f3e9-4f01-8253-fa90515daa53-tls-certs" pod "4419e8cb-f3e9-4f01-8253-fa90515daa53" (UID: "4419e8cb-f3e9-4f01-8253-fa90515daa53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:12.360755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.360732 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-dshm" pod "4419e8cb-f3e9-4f01-8253-fa90515daa53" (UID: "4419e8cb-f3e9-4f01-8253-fa90515daa53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:24:12.361230 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.361213 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4419e8cb-f3e9-4f01-8253-fa90515daa53-kube-api-access-gb9p2" pod "4419e8cb-f3e9-4f01-8253-fa90515daa53" (UID: "4419e8cb-f3e9-4f01-8253-fa90515daa53"). InnerVolumeSpecName "kube-api-access-gb9p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:24:12.373254 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.373235 4981 generic.go:356] "Generic (PLEG): container finished" podID="4419e8cb-f3e9-4f01-8253-fa90515daa53" containerID="50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681" exitCode=0 Nov 01 23:24:12.388596 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.373277 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" Nov 01 23:24:12.413269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.373292 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" event={"ID":"4419e8cb-f3e9-4f01-8253-fa90515daa53","Type":"ContainerDied","Data":"50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681"} Nov 01 23:24:12.413269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.397884 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" event={"ID":"4419e8cb-f3e9-4f01-8253-fa90515daa53","Type":"ContainerDied","Data":"975503d44777f8c594063cbd8fac9b8805cfbfbeb372c37834f28ac01dd7f7b1"} Nov 01 23:24:12.413269 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.397923 4981 scope.go:122] "RemoveContainer" containerID="50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681" Nov 01 23:24:12.456463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.456353 4981 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-home\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:12.456463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.456379 4981 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-model-cache\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:12.456463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.456409 4981 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-tmp-dir\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:12.456463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.456420 4981 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4419e8cb-f3e9-4f01-8253-fa90515daa53-dshm\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:12.456463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.456428 4981 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4419e8cb-f3e9-4f01-8253-fa90515daa53-tls-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:12.456463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.456437 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9p2\" (UniqueName: \"kubernetes.io/projected/4419e8cb-f3e9-4f01-8253-fa90515daa53-kube-api-access-gb9p2\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:12.459722 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.459662 4981 scope.go:122] "RemoveContainer" containerID="50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681" Nov 01 23:24:12.460710 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:24:12.460042 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681\": container with ID starting with 50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681 not found: ID does not exist" containerID="50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681" Nov 01 23:24:12.460710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.460074 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681"} err="failed to get container status \"50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681\": rpc error: code = NotFound desc = could not find container \"50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681\": container with ID starting with 50c753c3da18e9ca327499c7c78044b567c3e36333fe64e558262ab4a5bf4681 not found: ID does not exist" Nov 01 23:24:12.490609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.489643 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p"] Nov 01 23:24:12.506598 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.491519 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p"] Nov 01 23:24:12.746883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.746848 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4419e8cb-f3e9-4f01-8253-fa90515daa53" path="/var/lib/kubelet/pods/4419e8cb-f3e9-4f01-8253-fa90515daa53/volumes" Nov 01 23:24:12.747739 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:12.747716 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" path="/var/lib/kubelet/pods/fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d/volumes" Nov 01 23:24:13.115762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:13.115660 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-d660b8b1-kserve-dbc7d6fcc-dg77p" podUID="4419e8cb-f3e9-4f01-8253-fa90515daa53" containerName="main" probeResult="failure" output="Get \"https://10.134.1.151:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 23:24:14.528788 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:24:14.528746 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:24:16.190600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:16.187950 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:20.648286 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648250 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2"] Nov 01 23:24:20.648766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648709 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4419e8cb-f3e9-4f01-8253-fa90515daa53" containerName="main" Nov 01 23:24:20.648766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648721 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4419e8cb-f3e9-4f01-8253-fa90515daa53" containerName="main" Nov 01 23:24:20.648766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648737 4981 cpu_manager.go:414] "containerMap: removing container" podUID="fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" containerName="main" Nov 01 23:24:20.648766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648743 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" containerName="main" Nov 01 23:24:20.648766 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648765 4981 cpu_manager.go:414] "containerMap: removing container" podUID="86452723-09a0-4e7e-8f6c-21c2c93a3439" containerName="revoke-keys" Nov 01 23:24:20.648981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648771 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86452723-09a0-4e7e-8f6c-21c2c93a3439" containerName="revoke-keys" Nov 01 23:24:20.648981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648837 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="86452723-09a0-4e7e-8f6c-21c2c93a3439" containerName="revoke-keys" Nov 01 23:24:20.648981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648848 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4419e8cb-f3e9-4f01-8253-fa90515daa53" containerName="main" Nov 01 23:24:20.648981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.648856 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa09d70f-c98e-4f6c-970a-2bcae3c3aa5d" containerName="main" Nov 01 23:24:20.653693 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.653671 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.658319 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.658295 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:24:20.658604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.658507 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 23:24:20.658604 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.658547 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vwt76\"" Nov 01 23:24:20.658790 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.658695 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:24:20.659688 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.659640 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2"] Nov 01 23:24:20.786246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.786209 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.786477 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.786257 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmjr\" (UniqueName: \"kubernetes.io/projected/0869d024-1d26-415b-b8a8-4d71ce3dee2d-kube-api-access-dwmjr\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.786477 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.786321 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.887562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.887525 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.887816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.887658 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.887816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.887695 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmjr\" (UniqueName: \"kubernetes.io/projected/0869d024-1d26-415b-b8a8-4d71ce3dee2d-kube-api-access-dwmjr\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.888275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.888246 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.889993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.889970 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.895728 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.895706 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmjr\" (UniqueName: \"kubernetes.io/projected/0869d024-1d26-415b-b8a8-4d71ce3dee2d-kube-api-access-dwmjr\") pod \"maas-api-revoke-sub-97012f4d-k8gz2\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:20.970375 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:20.970330 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:21.242849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:21.242824 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2"] Nov 01 23:24:21.244955 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:24:21.244925 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0869d024_1d26_415b_b8a8_4d71ce3dee2d.slice/crio-795532f04f0e67ad8a8692e1388a014d2b1e4100ff450562c6d7eee9adc48c72 WatchSource:0}: Error finding container 795532f04f0e67ad8a8692e1388a014d2b1e4100ff450562c6d7eee9adc48c72: Status 404 returned error can't find the container with id 795532f04f0e67ad8a8692e1388a014d2b1e4100ff450562c6d7eee9adc48c72 Nov 01 23:24:21.469417 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:21.469383 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" event={"ID":"0869d024-1d26-415b-b8a8-4d71ce3dee2d","Type":"ContainerStarted","Data":"795532f04f0e67ad8a8692e1388a014d2b1e4100ff450562c6d7eee9adc48c72"} Nov 01 23:24:22.475938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:22.475903 4981 generic.go:356] "Generic (PLEG): container finished" podID="0869d024-1d26-415b-b8a8-4d71ce3dee2d" containerID="87eb32ffd775247504df4314893ecb285b9ab896ca9b9a41689f7d5825e5e9b6" exitCode=0 Nov 01 23:24:22.476369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:22.475938 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" event={"ID":"0869d024-1d26-415b-b8a8-4d71ce3dee2d","Type":"ContainerDied","Data":"87eb32ffd775247504df4314893ecb285b9ab896ca9b9a41689f7d5825e5e9b6"} Nov 01 23:24:23.628186 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.628158 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:23.718144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.718101 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0869d024-1d26-415b-b8a8-4d71ce3dee2d-kube-api-access-dwmjr\" (UniqueName: \"kubernetes.io/projected/0869d024-1d26-415b-b8a8-4d71ce3dee2d-kube-api-access-dwmjr\") pod \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " Nov 01 23:24:23.718379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.718164 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-xks-ca\") pod \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " Nov 01 23:24:23.718379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.718207 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-openshift-ca\") pod \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\" (UID: \"0869d024-1d26-415b-b8a8-4d71ce3dee2d\") " Nov 01 23:24:23.718683 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.718655 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-openshift-ca" pod "0869d024-1d26-415b-b8a8-4d71ce3dee2d" (UID: "0869d024-1d26-415b-b8a8-4d71ce3dee2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:24:23.720415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.720394 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-xks-ca" pod "0869d024-1d26-415b-b8a8-4d71ce3dee2d" (UID: "0869d024-1d26-415b-b8a8-4d71ce3dee2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:23.720491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.720414 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0869d024-1d26-415b-b8a8-4d71ce3dee2d-kube-api-access-dwmjr" pod "0869d024-1d26-415b-b8a8-4d71ce3dee2d" (UID: "0869d024-1d26-415b-b8a8-4d71ce3dee2d"). InnerVolumeSpecName "kube-api-access-dwmjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:24:23.819917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.819819 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmjr\" (UniqueName: \"kubernetes.io/projected/0869d024-1d26-415b-b8a8-4d71ce3dee2d-kube-api-access-dwmjr\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:23.819917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.819855 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-xks-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:23.819917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:23.819866 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0869d024-1d26-415b-b8a8-4d71ce3dee2d-maas-api-openshift-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:24.486712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:24.486680 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" Nov 01 23:24:24.486916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:24.486704 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2" event={"ID":"0869d024-1d26-415b-b8a8-4d71ce3dee2d","Type":"ContainerDied","Data":"795532f04f0e67ad8a8692e1388a014d2b1e4100ff450562c6d7eee9adc48c72"} Nov 01 23:24:24.486916 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:24.486756 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="795532f04f0e67ad8a8692e1388a014d2b1e4100ff450562c6d7eee9adc48c72" Nov 01 23:24:32.814802 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.814768 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x"] Nov 01 23:24:32.815280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.815195 4981 cpu_manager.go:414] "containerMap: removing container" podUID="0869d024-1d26-415b-b8a8-4d71ce3dee2d" containerName="revoke-keys" Nov 01 23:24:32.815280 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.815205 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0869d024-1d26-415b-b8a8-4d71ce3dee2d" containerName="revoke-keys" Nov 01 23:24:32.815363 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.815302 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="0869d024-1d26-415b-b8a8-4d71ce3dee2d" containerName="revoke-keys" Nov 01 23:24:32.818540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.818510 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:32.830521 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.830490 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 23:24:32.830773 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.830752 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 23:24:32.830882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.830806 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 23:24:32.830882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.830845 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-57phm\"" Nov 01 23:24:32.832076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.832049 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 23:24:32.833858 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.833835 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x"] Nov 01 23:24:32.837169 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.837147 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7587df7975-7qwpp"] Nov 01 23:24:32.840932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.840912 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:32.848310 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.848281 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7587df7975-7qwpp"] Nov 01 23:24:32.911781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.911739 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-maas-api-tls\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:32.912034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.911804 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27322fe7-25cf-4c7a-8733-8a0150346e19-openshift-service-ca\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:32.912034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.911935 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtlt\" (UniqueName: \"kubernetes.io/projected/b8ce96a2-e0f2-4128-997c-6897dde1c211-kube-api-access-xbtlt\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:32.912034 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.911986 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-metrics-certs\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:32.912265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.912065 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllj5\" (UniqueName: \"kubernetes.io/projected/27322fe7-25cf-4c7a-8733-8a0150346e19-kube-api-access-mllj5\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:32.912265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.912143 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27322fe7-25cf-4c7a-8733-8a0150346e19-maas-api-tls\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:32.912265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.912191 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27322fe7-25cf-4c7a-8733-8a0150346e19-metrics-certs\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:32.912265 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:32.912227 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce96a2-e0f2-4128-997c-6897dde1c211-openshift-service-ca\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.013868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.013827 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27322fe7-25cf-4c7a-8733-8a0150346e19-maas-api-tls\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.013868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.013873 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27322fe7-25cf-4c7a-8733-8a0150346e19-metrics-certs\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.014131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.013996 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce96a2-e0f2-4128-997c-6897dde1c211-openshift-service-ca\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.014131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.014077 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-maas-api-tls\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.014258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.014158 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27322fe7-25cf-4c7a-8733-8a0150346e19-openshift-service-ca\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.014258 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.014214 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtlt\" (UniqueName: \"kubernetes.io/projected/b8ce96a2-e0f2-4128-997c-6897dde1c211-kube-api-access-xbtlt\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.014383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.014275 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-metrics-certs\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.014383 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.014309 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllj5\" (UniqueName: \"kubernetes.io/projected/27322fe7-25cf-4c7a-8733-8a0150346e19-kube-api-access-mllj5\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.014850 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.014825 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce96a2-e0f2-4128-997c-6897dde1c211-openshift-service-ca\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.015349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.015311 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27322fe7-25cf-4c7a-8733-8a0150346e19-openshift-service-ca\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.016558 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.016524 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27322fe7-25cf-4c7a-8733-8a0150346e19-metrics-certs\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.016792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.016771 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-maas-api-tls\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.016862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.016772 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-metrics-certs\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.016862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.016779 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27322fe7-25cf-4c7a-8733-8a0150346e19-maas-api-tls\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.022147 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.022122 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtlt\" (UniqueName: \"kubernetes.io/projected/b8ce96a2-e0f2-4128-997c-6897dde1c211-kube-api-access-xbtlt\") pod \"maas-api-dcc8544c-n2q5x\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.022240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.022160 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllj5\" (UniqueName: \"kubernetes.io/projected/27322fe7-25cf-4c7a-8733-8a0150346e19-kube-api-access-mllj5\") pod \"maas-api-7587df7975-7qwpp\" (UID: \"27322fe7-25cf-4c7a-8733-8a0150346e19\") " pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.133616 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.133507 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:33.159248 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.159217 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:33.418803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.418773 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x"] Nov 01 23:24:33.420250 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:24:33.420221 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ce96a2_e0f2_4128_997c_6897dde1c211.slice/crio-39a1a4cf377d2f053db178b2eac8b8d04be79138793c653a34dbbc4499889c1a WatchSource:0}: Error finding container 39a1a4cf377d2f053db178b2eac8b8d04be79138793c653a34dbbc4499889c1a: Status 404 returned error can't find the container with id 39a1a4cf377d2f053db178b2eac8b8d04be79138793c653a34dbbc4499889c1a Nov 01 23:24:33.421724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.421706 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:24:33.449482 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.449459 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7587df7975-7qwpp"] Nov 01 23:24:33.451321 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:24:33.451260 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27322fe7_25cf_4c7a_8733_8a0150346e19.slice/crio-e0e197690bcb1461b385f80eaea2503575682eccdaa4eef122b88cb4d43bca0c WatchSource:0}: Error finding container e0e197690bcb1461b385f80eaea2503575682eccdaa4eef122b88cb4d43bca0c: Status 404 returned error can't find the container with id e0e197690bcb1461b385f80eaea2503575682eccdaa4eef122b88cb4d43bca0c Nov 01 23:24:33.532320 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.532280 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" event={"ID":"27322fe7-25cf-4c7a-8733-8a0150346e19","Type":"ContainerStarted","Data":"e0e197690bcb1461b385f80eaea2503575682eccdaa4eef122b88cb4d43bca0c"} Nov 01 23:24:33.533530 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:33.533505 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" event={"ID":"b8ce96a2-e0f2-4128-997c-6897dde1c211","Type":"ContainerStarted","Data":"39a1a4cf377d2f053db178b2eac8b8d04be79138793c653a34dbbc4499889c1a"} Nov 01 23:24:34.540637 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:34.540568 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" event={"ID":"27322fe7-25cf-4c7a-8733-8a0150346e19","Type":"ContainerStarted","Data":"431eda70483d2205015eefc284c8e49d097e73215596e2d443c4b4fd50ae3719"} Nov 01 23:24:34.541122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:34.540774 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:34.542156 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:34.542129 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" event={"ID":"b8ce96a2-e0f2-4128-997c-6897dde1c211","Type":"ContainerStarted","Data":"3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af"} Nov 01 23:24:34.542296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:34.542174 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:34.558508 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:34.558458 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" podStartSLOduration=2.092493909 podStartE2EDuration="2.558441665s" totalImagesPullingTime="465.947756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:34.557220734 +0000 UTC m=+3639.985098667" watchObservedRunningTime="2026-11-01 23:24:34.558441665 +0000 UTC m=+3639.986319593" Nov 01 23:24:34.574050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:34.573994 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" podStartSLOduration=2.09706998 podStartE2EDuration="2.573976719s" totalImagesPullingTime="476.906739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:34.573119225 +0000 UTC m=+3640.000997158" watchObservedRunningTime="2026-11-01 23:24:34.573976719 +0000 UTC m=+3640.001854652" Nov 01 23:24:39.021314 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:39.021214 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw"] Nov 01 23:24:39.023349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:39.023324 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d87mw"] Nov 01 23:24:39.470144 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:39.470114 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:39.900471 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:39.900390 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:39.996248 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:39.996210 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:40.552839 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.552810 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7587df7975-7qwpp" Nov 01 23:24:40.553361 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.553234 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:40.612303 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.612264 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x"] Nov 01 23:24:40.612513 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.612481 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" podUID="b8ce96a2-e0f2-4128-997c-6897dde1c211" containerName="maas-api" containerID="cri-o://3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af" gracePeriod=30 Nov 01 23:24:40.734626 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.734596 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f7747a-8503-47ae-90a5-4b61394d9af5" path="/var/lib/kubelet/pods/00f7747a-8503-47ae-90a5-4b61394d9af5/volumes" Nov 01 23:24:40.867680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.867653 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:40.996510 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.996465 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-metrics-certs\") pod \"b8ce96a2-e0f2-4128-997c-6897dde1c211\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " Nov 01 23:24:40.996735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.996561 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ce96a2-e0f2-4128-997c-6897dde1c211-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce96a2-e0f2-4128-997c-6897dde1c211-openshift-service-ca\") pod \"b8ce96a2-e0f2-4128-997c-6897dde1c211\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " Nov 01 23:24:40.996735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.996636 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-maas-api-tls\") pod \"b8ce96a2-e0f2-4128-997c-6897dde1c211\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " Nov 01 23:24:40.996735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.996671 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ce96a2-e0f2-4128-997c-6897dde1c211-kube-api-access-xbtlt\" (UniqueName: \"kubernetes.io/projected/b8ce96a2-e0f2-4128-997c-6897dde1c211-kube-api-access-xbtlt\") pod \"b8ce96a2-e0f2-4128-997c-6897dde1c211\" (UID: \"b8ce96a2-e0f2-4128-997c-6897dde1c211\") " Nov 01 23:24:40.996985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.996955 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ce96a2-e0f2-4128-997c-6897dde1c211-openshift-service-ca" pod "b8ce96a2-e0f2-4128-997c-6897dde1c211" (UID: "b8ce96a2-e0f2-4128-997c-6897dde1c211"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:24:40.998913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.998881 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-maas-api-tls" pod "b8ce96a2-e0f2-4128-997c-6897dde1c211" (UID: "b8ce96a2-e0f2-4128-997c-6897dde1c211"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:40.999307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.999261 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-metrics-certs" pod "b8ce96a2-e0f2-4128-997c-6897dde1c211" (UID: "b8ce96a2-e0f2-4128-997c-6897dde1c211"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:24:40.999307 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:40.999282 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ce96a2-e0f2-4128-997c-6897dde1c211-kube-api-access-xbtlt" pod "b8ce96a2-e0f2-4128-997c-6897dde1c211" (UID: "b8ce96a2-e0f2-4128-997c-6897dde1c211"). InnerVolumeSpecName "kube-api-access-xbtlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:24:41.098131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.098042 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:41.098131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.098077 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtlt\" (UniqueName: \"kubernetes.io/projected/b8ce96a2-e0f2-4128-997c-6897dde1c211-kube-api-access-xbtlt\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:41.098131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.098088 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8ce96a2-e0f2-4128-997c-6897dde1c211-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:41.098131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.098097 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8ce96a2-e0f2-4128-997c-6897dde1c211-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:24:41.582205 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.582168 4981 generic.go:356] "Generic (PLEG): container finished" podID="b8ce96a2-e0f2-4128-997c-6897dde1c211" containerID="3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af" exitCode=0 Nov 01 23:24:41.582699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.582218 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" Nov 01 23:24:41.582699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.582257 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" event={"ID":"b8ce96a2-e0f2-4128-997c-6897dde1c211","Type":"ContainerDied","Data":"3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af"} Nov 01 23:24:41.582699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.582309 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x" event={"ID":"b8ce96a2-e0f2-4128-997c-6897dde1c211","Type":"ContainerDied","Data":"39a1a4cf377d2f053db178b2eac8b8d04be79138793c653a34dbbc4499889c1a"} Nov 01 23:24:41.582699 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.582337 4981 scope.go:122] "RemoveContainer" containerID="3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af" Nov 01 23:24:41.592259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.592243 4981 scope.go:122] "RemoveContainer" containerID="3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af" Nov 01 23:24:41.592510 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:24:41.592492 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af\": container with ID starting with 3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af not found: ID does not exist" containerID="3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af" Nov 01 23:24:41.592559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.592519 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af"} err="failed to get container status \"3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af\": rpc error: code = NotFound desc = could not find container \"3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af\": container with ID starting with 3dca14ac165828846a28fe0238da42e2a43a20de3c3d933c8dea8c88ebc8a7af not found: ID does not exist" Nov 01 23:24:41.605600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.605530 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x"] Nov 01 23:24:41.607367 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:41.607347 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dcc8544c-n2q5x"] Nov 01 23:24:42.732912 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:42.732878 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ce96a2-e0f2-4128-997c-6897dde1c211" path="/var/lib/kubelet/pods/b8ce96a2-e0f2-4128-997c-6897dde1c211/volumes" Nov 01 23:24:42.914165 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:42.914128 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:43.017605 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:43.017488 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:44.118257 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.118220 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs"] Nov 01 23:24:44.118979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.118953 4981 cpu_manager.go:414] "containerMap: removing container" podUID="b8ce96a2-e0f2-4128-997c-6897dde1c211" containerName="maas-api" Nov 01 23:24:44.118979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.118980 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ce96a2-e0f2-4128-997c-6897dde1c211" containerName="maas-api" Nov 01 23:24:44.119150 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.119110 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ce96a2-e0f2-4128-997c-6897dde1c211" containerName="maas-api" Nov 01 23:24:44.124023 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.123995 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.124675 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.124639 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2817d29-0612-4dc3-b418-0e3850ff546f-openshift-service-ca\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.124796 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.124721 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-maas-api-tls\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.124917 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.124892 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-metrics-certs\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.124962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.124948 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w99\" (UniqueName: \"kubernetes.io/projected/c2817d29-0612-4dc3-b418-0e3850ff546f-kube-api-access-89w99\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.126890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.126864 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:44.141341 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.141310 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-417a41eb-serving-cert\"" Nov 01 23:24:44.141552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.141530 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs"] Nov 01 23:24:44.191520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.191490 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v"] Nov 01 23:24:44.195304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.195280 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.208589 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.208550 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v"] Nov 01 23:24:44.208755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.208709 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-417a41eb\"" Nov 01 23:24:44.215075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.215051 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-417a41eb-dockercfg-h7g8x\"" Nov 01 23:24:44.226538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.226495 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-metrics-certs\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.226749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.226544 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w99\" (UniqueName: \"kubernetes.io/projected/c2817d29-0612-4dc3-b418-0e3850ff546f-kube-api-access-89w99\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.226749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.226611 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3778ce27-143a-4f30-b67a-b81b84f863d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.226749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.226652 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2817d29-0612-4dc3-b418-0e3850ff546f-openshift-service-ca\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.226749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.226681 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpqmc\" (UniqueName: \"kubernetes.io/projected/3778ce27-143a-4f30-b67a-b81b84f863d3-kube-api-access-zpqmc\") pod \"payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.226749 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.226736 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-maas-api-tls\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.227894 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.227866 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2817d29-0612-4dc3-b418-0e3850ff546f-openshift-service-ca\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.229071 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.229051 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-metrics-certs\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.229181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.229164 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-maas-api-tls\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.235603 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.235552 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w99\" (UniqueName: \"kubernetes.io/projected/c2817d29-0612-4dc3-b418-0e3850ff546f-kube-api-access-89w99\") pod \"maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.255126 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.255092 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df"] Nov 01 23:24:44.259148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.259122 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.261646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.261621 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:44.270337 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.270305 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df"] Nov 01 23:24:44.327731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.327691 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jsvx\" (UniqueName: \"kubernetes.io/projected/93d281ec-b985-431d-bda9-35dfb349bc86-kube-api-access-5jsvx\") pod \"payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.327949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.327749 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d281ec-b985-431d-bda9-35dfb349bc86-plugins-config-volume\") pod \"payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.327949 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.327911 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3778ce27-143a-4f30-b67a-b81b84f863d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.328086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.327974 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpqmc\" (UniqueName: \"kubernetes.io/projected/3778ce27-143a-4f30-b67a-b81b84f863d3-kube-api-access-zpqmc\") pod \"payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.328879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.328852 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3778ce27-143a-4f30-b67a-b81b84f863d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.338152 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.338116 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpqmc\" (UniqueName: \"kubernetes.io/projected/3778ce27-143a-4f30-b67a-b81b84f863d3-kube-api-access-zpqmc\") pod \"payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.367833 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.367782 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:44.428809 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.428771 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jsvx\" (UniqueName: \"kubernetes.io/projected/93d281ec-b985-431d-bda9-35dfb349bc86-kube-api-access-5jsvx\") pod \"payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.429038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.428850 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d281ec-b985-431d-bda9-35dfb349bc86-plugins-config-volume\") pod \"payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.429691 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.429668 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d281ec-b985-431d-bda9-35dfb349bc86-plugins-config-volume\") pod \"payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.437826 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.437789 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jsvx\" (UniqueName: \"kubernetes.io/projected/93d281ec-b985-431d-bda9-35dfb349bc86-kube-api-access-5jsvx\") pod \"payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.443643 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.443618 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:44.511220 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.511184 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:44.536116 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:24:44.536083 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:24:44.577354 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.574983 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:44.730119 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:24:44.730076 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2817d29_0612_4dc3_b418_0e3850ff546f.slice/crio-170ef5e8561e693d82a2e3697cbb0a889456def1feaf004f0d04aee37b4d6ad4 WatchSource:0}: Error finding container 170ef5e8561e693d82a2e3697cbb0a889456def1feaf004f0d04aee37b4d6ad4: Status 404 returned error can't find the container with id 170ef5e8561e693d82a2e3697cbb0a889456def1feaf004f0d04aee37b4d6ad4 Nov 01 23:24:44.734001 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.733980 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs"] Nov 01 23:24:44.792072 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.792047 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v"] Nov 01 23:24:44.793405 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:24:44.793382 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3778ce27_143a_4f30_b67a_b81b84f863d3.slice/crio-70e34c91ae35d6626b750fd33708829985246be4d3b17b1d1de422e871d417fb WatchSource:0}: Error finding container 70e34c91ae35d6626b750fd33708829985246be4d3b17b1d1de422e871d417fb: Status 404 returned error can't find the container with id 70e34c91ae35d6626b750fd33708829985246be4d3b17b1d1de422e871d417fb Nov 01 23:24:44.856681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.856655 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df"] Nov 01 23:24:44.859124 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:24:44.859100 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d281ec_b985_431d_bda9_35dfb349bc86.slice/crio-8c4049897efbc06a346dcefeae8771abe3a909d17664dce683256e511274c9fe WatchSource:0}: Error finding container 8c4049897efbc06a346dcefeae8771abe3a909d17664dce683256e511274c9fe: Status 404 returned error can't find the container with id 8c4049897efbc06a346dcefeae8771abe3a909d17664dce683256e511274c9fe Nov 01 23:24:44.947973 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:44.947940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:45.553446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.553372 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:45.606002 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.605960 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" event={"ID":"c2817d29-0612-4dc3-b418-0e3850ff546f","Type":"ContainerStarted","Data":"0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b"} Nov 01 23:24:45.606196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.606030 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" event={"ID":"c2817d29-0612-4dc3-b418-0e3850ff546f","Type":"ContainerStarted","Data":"170ef5e8561e693d82a2e3697cbb0a889456def1feaf004f0d04aee37b4d6ad4"} Nov 01 23:24:45.606196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.606086 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:45.607785 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.607761 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" event={"ID":"93d281ec-b985-431d-bda9-35dfb349bc86","Type":"ContainerStarted","Data":"400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4"} Nov 01 23:24:45.607941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.607800 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" event={"ID":"93d281ec-b985-431d-bda9-35dfb349bc86","Type":"ContainerStarted","Data":"8c4049897efbc06a346dcefeae8771abe3a909d17664dce683256e511274c9fe"} Nov 01 23:24:45.607941 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.607853 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:45.609512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.609425 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" event={"ID":"3778ce27-143a-4f30-b67a-b81b84f863d3","Type":"ContainerStarted","Data":"4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5"} Nov 01 23:24:45.609512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.609479 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" event={"ID":"3778ce27-143a-4f30-b67a-b81b84f863d3","Type":"ContainerStarted","Data":"70e34c91ae35d6626b750fd33708829985246be4d3b17b1d1de422e871d417fb"} Nov 01 23:24:45.609512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.609510 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:45.627345 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.627286 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" podStartSLOduration=1.168199429 podStartE2EDuration="1.627272768s" totalImagesPullingTime="459.073339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:45.623321693 +0000 UTC m=+3651.051199623" watchObservedRunningTime="2026-11-01 23:24:45.627272768 +0000 UTC m=+3651.055150699" Nov 01 23:24:45.653618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.653547 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" podStartSLOduration=1.6535329989999998 podStartE2EDuration="1.653532999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:45.651625732 +0000 UTC m=+3651.079503663" watchObservedRunningTime="2026-11-01 23:24:45.653532999 +0000 UTC m=+3651.081410930" Nov 01 23:24:45.682229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:45.682169 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" podStartSLOduration=1.682152485 podStartE2EDuration="1.682152485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:24:45.681242108 +0000 UTC m=+3651.109120040" watchObservedRunningTime="2026-11-01 23:24:45.682152485 +0000 UTC m=+3651.110030416" Nov 01 23:24:46.251923 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:46.251889 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:47.060327 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:47.060279 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:47.773862 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:47.773828 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:48.480726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:48.480681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:49.183890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:49.183854 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:49.885560 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:49.885524 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:50.673724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:50.673691 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:51.372719 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:51.372681 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:51.619870 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:51.619836 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:24:52.228712 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:52.228656 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:52.970844 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:52.970806 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:53.733255 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:53.733216 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:54.475004 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:54.474976 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:55.070059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:55.070018 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:55.822250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:55.822211 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:56.536281 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:56.536247 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:56.617272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:56.617244 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:24:56.617489 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:56.617401 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:24:57.165263 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:57.165223 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:57.971086 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:57.971048 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:58.737962 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:58.737907 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:24:59.504207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:24:59.504167 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:00.182969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:00.182918 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:00.929204 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:00.929168 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:01.772979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:01.772922 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:02.465419 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:02.465381 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:03.076357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:03.076316 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:03.642876 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:03.642841 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df"] Nov 01 23:25:03.655302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:03.655268 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v"] Nov 01 23:25:03.703998 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:03.703962 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:03.801976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:03.801941 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:03.901803 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:03.901718 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:04.444026 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.443993 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs"] Nov 01 23:25:04.444562 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.444251 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" podUID="c2817d29-0612-4dc3-b418-0e3850ff546f" containerName="maas-api" containerID="cri-o://0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b" gracePeriod=30 Nov 01 23:25:04.487379 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.487347 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:04.626818 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.626775 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:04.713909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.713882 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:25:04.733350 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.733320 4981 generic.go:356] "Generic (PLEG): container finished" podID="c2817d29-0612-4dc3-b418-0e3850ff546f" containerID="0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b" exitCode=0 Nov 01 23:25:04.737463 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.737434 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" Nov 01 23:25:04.739623 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.739596 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" event={"ID":"c2817d29-0612-4dc3-b418-0e3850ff546f","Type":"ContainerDied","Data":"0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b"} Nov 01 23:25:04.739834 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.739645 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs" event={"ID":"c2817d29-0612-4dc3-b418-0e3850ff546f","Type":"ContainerDied","Data":"170ef5e8561e693d82a2e3697cbb0a889456def1feaf004f0d04aee37b4d6ad4"} Nov 01 23:25:04.739834 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.739685 4981 scope.go:122] "RemoveContainer" containerID="0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b" Nov 01 23:25:04.750631 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.750606 4981 scope.go:122] "RemoveContainer" containerID="0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b" Nov 01 23:25:04.750982 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:25:04.750957 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b\": container with ID starting with 0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b not found: ID does not exist" containerID="0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b" Nov 01 23:25:04.751092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.750996 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b"} err="failed to get container status \"0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b\": rpc error: code = NotFound desc = could not find container \"0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b\": container with ID starting with 0575ff2ab2def402b9ba923755b31a040eace944f82c92a721ee77ea9e86144b not found: ID does not exist" Nov 01 23:25:04.752010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.751973 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:04.809724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.809682 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2817d29-0612-4dc3-b418-0e3850ff546f-kube-api-access-89w99\" (UniqueName: \"kubernetes.io/projected/c2817d29-0612-4dc3-b418-0e3850ff546f-kube-api-access-89w99\") pod \"c2817d29-0612-4dc3-b418-0e3850ff546f\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " Nov 01 23:25:04.809961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.809775 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-metrics-certs\") pod \"c2817d29-0612-4dc3-b418-0e3850ff546f\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " Nov 01 23:25:04.809961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.809826 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-maas-api-tls\") pod \"c2817d29-0612-4dc3-b418-0e3850ff546f\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " Nov 01 23:25:04.809961 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.809860 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2817d29-0612-4dc3-b418-0e3850ff546f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2817d29-0612-4dc3-b418-0e3850ff546f-openshift-service-ca\") pod \"c2817d29-0612-4dc3-b418-0e3850ff546f\" (UID: \"c2817d29-0612-4dc3-b418-0e3850ff546f\") " Nov 01 23:25:04.810369 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.810337 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2817d29-0612-4dc3-b418-0e3850ff546f-openshift-service-ca" pod "c2817d29-0612-4dc3-b418-0e3850ff546f" (UID: "c2817d29-0612-4dc3-b418-0e3850ff546f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:25:04.811981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.811951 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-metrics-certs" pod "c2817d29-0612-4dc3-b418-0e3850ff546f" (UID: "c2817d29-0612-4dc3-b418-0e3850ff546f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:25:04.811981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.811969 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-maas-api-tls" pod "c2817d29-0612-4dc3-b418-0e3850ff546f" (UID: "c2817d29-0612-4dc3-b418-0e3850ff546f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:25:04.812124 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.811976 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2817d29-0612-4dc3-b418-0e3850ff546f-kube-api-access-89w99" pod "c2817d29-0612-4dc3-b418-0e3850ff546f" (UID: "c2817d29-0612-4dc3-b418-0e3850ff546f"). InnerVolumeSpecName "kube-api-access-89w99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:25:04.911373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.911339 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:04.911373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.911369 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2817d29-0612-4dc3-b418-0e3850ff546f-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:04.911373 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.911379 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2817d29-0612-4dc3-b418-0e3850ff546f-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:04.911662 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:04.911388 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89w99\" (UniqueName: \"kubernetes.io/projected/c2817d29-0612-4dc3-b418-0e3850ff546f-kube-api-access-89w99\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:05.060836 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:05.060556 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs"] Nov 01 23:25:05.063412 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:05.063385 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-417a41eb-5ff48c5846-zlkgs"] Nov 01 23:25:06.732851 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:06.732816 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2817d29-0612-4dc3-b418-0e3850ff546f" path="/var/lib/kubelet/pods/c2817d29-0612-4dc3-b418-0e3850ff546f/volumes" Nov 01 23:25:08.643959 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.643916 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" podUID="93d281ec-b985-431d-bda9-35dfb349bc86" containerName="payload-processing" containerID="cri-o://400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4" gracePeriod=25 Nov 01 23:25:08.655539 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.655510 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" podUID="3778ce27-143a-4f30-b67a-b81b84f863d3" containerName="payload-pre-processing" containerID="cri-o://4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5" gracePeriod=25 Nov 01 23:25:08.912252 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.912182 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:25:08.942948 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.942919 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:25:08.950692 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.950647 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d281ec-b985-431d-bda9-35dfb349bc86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d281ec-b985-431d-bda9-35dfb349bc86-plugins-config-volume\") pod \"93d281ec-b985-431d-bda9-35dfb349bc86\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " Nov 01 23:25:08.950820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.950763 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d281ec-b985-431d-bda9-35dfb349bc86-kube-api-access-5jsvx\" (UniqueName: \"kubernetes.io/projected/93d281ec-b985-431d-bda9-35dfb349bc86-kube-api-access-5jsvx\") pod \"93d281ec-b985-431d-bda9-35dfb349bc86\" (UID: \"93d281ec-b985-431d-bda9-35dfb349bc86\") " Nov 01 23:25:08.951029 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.951005 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d281ec-b985-431d-bda9-35dfb349bc86-plugins-config-volume" pod "93d281ec-b985-431d-bda9-35dfb349bc86" (UID: "93d281ec-b985-431d-bda9-35dfb349bc86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:25:08.951153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.951135 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d281ec-b985-431d-bda9-35dfb349bc86-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:08.952913 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:08.952889 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d281ec-b985-431d-bda9-35dfb349bc86-kube-api-access-5jsvx" pod "93d281ec-b985-431d-bda9-35dfb349bc86" (UID: "93d281ec-b985-431d-bda9-35dfb349bc86"). InnerVolumeSpecName "kube-api-access-5jsvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:25:09.052059 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.052016 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3778ce27-143a-4f30-b67a-b81b84f863d3-kube-api-access-zpqmc\" (UniqueName: \"kubernetes.io/projected/3778ce27-143a-4f30-b67a-b81b84f863d3-kube-api-access-zpqmc\") pod \"3778ce27-143a-4f30-b67a-b81b84f863d3\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " Nov 01 23:25:09.052254 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.052106 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3778ce27-143a-4f30-b67a-b81b84f863d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3778ce27-143a-4f30-b67a-b81b84f863d3-plugins-config-volume\") pod \"3778ce27-143a-4f30-b67a-b81b84f863d3\" (UID: \"3778ce27-143a-4f30-b67a-b81b84f863d3\") " Nov 01 23:25:09.052392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.052380 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jsvx\" (UniqueName: \"kubernetes.io/projected/93d281ec-b985-431d-bda9-35dfb349bc86-kube-api-access-5jsvx\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:09.052546 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.052523 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3778ce27-143a-4f30-b67a-b81b84f863d3-plugins-config-volume" pod "3778ce27-143a-4f30-b67a-b81b84f863d3" (UID: "3778ce27-143a-4f30-b67a-b81b84f863d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:25:09.054276 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.054255 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3778ce27-143a-4f30-b67a-b81b84f863d3-kube-api-access-zpqmc" pod "3778ce27-143a-4f30-b67a-b81b84f863d3" (UID: "3778ce27-143a-4f30-b67a-b81b84f863d3"). InnerVolumeSpecName "kube-api-access-zpqmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:25:09.153105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.153066 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpqmc\" (UniqueName: \"kubernetes.io/projected/3778ce27-143a-4f30-b67a-b81b84f863d3-kube-api-access-zpqmc\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:09.153105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.153097 4981 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3778ce27-143a-4f30-b67a-b81b84f863d3-plugins-config-volume\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:25:09.760647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.760610 4981 generic.go:356] "Generic (PLEG): container finished" podID="93d281ec-b985-431d-bda9-35dfb349bc86" containerID="400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4" exitCode=0 Nov 01 23:25:09.760647 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.760636 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" Nov 01 23:25:09.761196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.760669 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" event={"ID":"93d281ec-b985-431d-bda9-35dfb349bc86","Type":"ContainerDied","Data":"400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4"} Nov 01 23:25:09.761196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.760722 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df" event={"ID":"93d281ec-b985-431d-bda9-35dfb349bc86","Type":"ContainerDied","Data":"8c4049897efbc06a346dcefeae8771abe3a909d17664dce683256e511274c9fe"} Nov 01 23:25:09.761196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.760752 4981 scope.go:122] "RemoveContainer" containerID="400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4" Nov 01 23:25:09.762483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.762459 4981 generic.go:356] "Generic (PLEG): container finished" podID="3778ce27-143a-4f30-b67a-b81b84f863d3" containerID="4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5" exitCode=0 Nov 01 23:25:09.762669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.762507 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" Nov 01 23:25:09.762669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.762516 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" event={"ID":"3778ce27-143a-4f30-b67a-b81b84f863d3","Type":"ContainerDied","Data":"4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5"} Nov 01 23:25:09.762669 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.762557 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v" event={"ID":"3778ce27-143a-4f30-b67a-b81b84f863d3","Type":"ContainerDied","Data":"70e34c91ae35d6626b750fd33708829985246be4d3b17b1d1de422e871d417fb"} Nov 01 23:25:09.772313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.772292 4981 scope.go:122] "RemoveContainer" containerID="400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4" Nov 01 23:25:09.772750 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:25:09.772724 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4\": container with ID starting with 400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4 not found: ID does not exist" containerID="400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4" Nov 01 23:25:09.772874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.772762 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4"} err="failed to get container status \"400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4\": rpc error: code = NotFound desc = could not find container \"400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4\": container with ID starting with 400169d1656e5e50e25f14266b3ae5f633aa35a641d6d4c219962be0ed74afa4 not found: ID does not exist" Nov 01 23:25:09.772874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.772785 4981 scope.go:122] "RemoveContainer" containerID="4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5" Nov 01 23:25:09.773701 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.773678 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:09.781519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.781504 4981 scope.go:122] "RemoveContainer" containerID="4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5" Nov 01 23:25:09.781785 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:25:09.781766 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5\": container with ID starting with 4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5 not found: ID does not exist" containerID="4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5" Nov 01 23:25:09.781861 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.781792 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5"} err="failed to get container status \"4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5\": rpc error: code = NotFound desc = could not find container \"4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5\": container with ID starting with 4baabb881ac2792a25819ceee9bfa2c470f4367a8ed170eb192e50ca6a76d6d5 not found: ID does not exist" Nov 01 23:25:09.789378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.789358 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df"] Nov 01 23:25:09.793062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.793042 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-417a41eb-7cd859cf69-9t6df"] Nov 01 23:25:09.804682 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.804660 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v"] Nov 01 23:25:09.805277 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:09.805260 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-417a41eb-5b86fbdd75-gjc4v"] Nov 01 23:25:10.733831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:10.733792 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3778ce27-143a-4f30-b67a-b81b84f863d3" path="/var/lib/kubelet/pods/3778ce27-143a-4f30-b67a-b81b84f863d3/volumes" Nov 01 23:25:10.734223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:10.734210 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d281ec-b985-431d-bda9-35dfb349bc86" path="/var/lib/kubelet/pods/93d281ec-b985-431d-bda9-35dfb349bc86/volumes" Nov 01 23:25:10.956077 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:10.956043 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:11.331807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.331762 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg"] Nov 01 23:25:11.332279 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332262 4981 cpu_manager.go:414] "containerMap: removing container" podUID="c2817d29-0612-4dc3-b418-0e3850ff546f" containerName="maas-api" Nov 01 23:25:11.332364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332281 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2817d29-0612-4dc3-b418-0e3850ff546f" containerName="maas-api" Nov 01 23:25:11.332364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332304 4981 cpu_manager.go:414] "containerMap: removing container" podUID="3778ce27-143a-4f30-b67a-b81b84f863d3" containerName="payload-pre-processing" Nov 01 23:25:11.332364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332309 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3778ce27-143a-4f30-b67a-b81b84f863d3" containerName="payload-pre-processing" Nov 01 23:25:11.332364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332318 4981 cpu_manager.go:414] "containerMap: removing container" podUID="93d281ec-b985-431d-bda9-35dfb349bc86" containerName="payload-processing" Nov 01 23:25:11.332364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332322 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d281ec-b985-431d-bda9-35dfb349bc86" containerName="payload-processing" Nov 01 23:25:11.332540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332384 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2817d29-0612-4dc3-b418-0e3850ff546f" containerName="maas-api" Nov 01 23:25:11.332540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332397 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="3778ce27-143a-4f30-b67a-b81b84f863d3" containerName="payload-pre-processing" Nov 01 23:25:11.332540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.332403 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d281ec-b985-431d-bda9-35dfb349bc86" containerName="payload-processing" Nov 01 23:25:11.337148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.337129 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.342511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.342476 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-ede84d-openshift-default-dockercfg-d592d\"" Nov 01 23:25:11.348387 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.347955 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg"] Nov 01 23:25:11.376196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376162 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-557ws\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-kube-api-access-557ws\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376216 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-certs\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376316 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-envoy\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376363 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-credential-socket\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376448 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-data\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376481 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-podinfo\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376507 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-socket\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376609 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-token\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.376681 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.376632 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b35ffa6-b273-494c-b80f-7a4602de25f6-istiod-ca-cert\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.383094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.383064 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:11.477364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477326 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-data\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477364 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477368 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-podinfo\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477392 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-socket\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477588 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-token\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477648 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b35ffa6-b273-494c-b80f-7a4602de25f6-istiod-ca-cert\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477713 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-557ws\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-kube-api-access-557ws\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477786 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-certs\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477840 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-data\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.477884 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477856 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-envoy\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.478131 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.477931 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-credential-socket\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.478203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.478140 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-certs\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.478203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.478180 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-socket\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.478344 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.478333 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-credential-socket\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.478408 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.478357 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b35ffa6-b273-494c-b80f-7a4602de25f6-istiod-ca-cert\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.480049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.480024 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-envoy\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.480149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.480098 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-podinfo\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.485983 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.485952 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-token\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.486104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.486035 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-557ws\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-kube-api-access-557ws\") pod \"e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.507726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.507698 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:11.656926 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.656830 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:11.937753 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.937722 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg"] Nov 01 23:25:11.939521 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:25:11.939491 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b35ffa6_b273_494c_b80f_7a4602de25f6.slice/crio-8a875a16c92a951b89056bf0deccfaf7afb9b5787e39c698d987d8f5aa343469 WatchSource:0}: Error finding container 8a875a16c92a951b89056bf0deccfaf7afb9b5787e39c698d987d8f5aa343469: Status 404 returned error can't find the container with id 8a875a16c92a951b89056bf0deccfaf7afb9b5787e39c698d987d8f5aa343469 Nov 01 23:25:11.941761 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.941730 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:25:11.941871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.941802 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:25:11.941871 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:11.941852 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:25:12.789149 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:12.789109 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" event={"ID":"4b35ffa6-b273-494c-b80f-7a4602de25f6","Type":"ContainerStarted","Data":"cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344"} Nov 01 23:25:12.789557 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:12.789168 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" event={"ID":"4b35ffa6-b273-494c-b80f-7a4602de25f6","Type":"ContainerStarted","Data":"8a875a16c92a951b89056bf0deccfaf7afb9b5787e39c698d987d8f5aa343469"} Nov 01 23:25:12.812628 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:12.812562 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podStartSLOduration=1.81254957 podStartE2EDuration="1.81254957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:25:12.809762171 +0000 UTC m=+3678.237640113" watchObservedRunningTime="2026-11-01 23:25:12.81254957 +0000 UTC m=+3678.240427500" Nov 01 23:25:13.657670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:13.657635 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:13.663094 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:13.663066 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:13.794552 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:13.794520 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:13.795559 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:13.795531 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:14.523816 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:25:14.523778 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:25:14.914018 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:14.913916 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:15.018459 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:15.018423 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:16.020148 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.020111 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj"] Nov 01 23:25:16.024268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.024246 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.029325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.029299 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-ede84d-serving-cert\"" Nov 01 23:25:16.039157 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.039128 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj"] Nov 01 23:25:16.076729 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.076678 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:16.127989 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.127944 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2167f126-c1a8-4410-9622-52fd2b0ff06e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.128196 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.128107 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tkc4\" (UniqueName: \"kubernetes.io/projected/2167f126-c1a8-4410-9622-52fd2b0ff06e-kube-api-access-4tkc4\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.128318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.128290 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.128386 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.128367 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.175175 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.175148 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:16.229063 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.229032 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tkc4\" (UniqueName: \"kubernetes.io/projected/2167f126-c1a8-4410-9622-52fd2b0ff06e-kube-api-access-4tkc4\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.229246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.229146 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.229246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.229189 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.229246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.229233 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2167f126-c1a8-4410-9622-52fd2b0ff06e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.230050 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.230020 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2167f126-c1a8-4410-9622-52fd2b0ff06e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.231953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.231926 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.232102 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.231969 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.244732 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.244702 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tkc4\" (UniqueName: \"kubernetes.io/projected/2167f126-c1a8-4410-9622-52fd2b0ff06e-kube-api-access-4tkc4\") pod \"maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.283927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.283825 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:16.339308 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.339269 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:16.617431 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.617399 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj"] Nov 01 23:25:16.618424 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:25:16.618400 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2167f126_c1a8_4410_9622_52fd2b0ff06e.slice/crio-ecfb29899d6fe6f42032954973e8ccca15d8cfa3cc1ea0c51827bb8b422b4b99 WatchSource:0}: Error finding container ecfb29899d6fe6f42032954973e8ccca15d8cfa3cc1ea0c51827bb8b422b4b99: Status 404 returned error can't find the container with id ecfb29899d6fe6f42032954973e8ccca15d8cfa3cc1ea0c51827bb8b422b4b99 Nov 01 23:25:16.809648 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.809607 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" event={"ID":"2167f126-c1a8-4410-9622-52fd2b0ff06e","Type":"ContainerStarted","Data":"ecfb29899d6fe6f42032954973e8ccca15d8cfa3cc1ea0c51827bb8b422b4b99"} Nov 01 23:25:16.969708 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:16.969673 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:17.663409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:17.663374 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:17.816653 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:17.816615 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" event={"ID":"2167f126-c1a8-4410-9622-52fd2b0ff06e","Type":"ContainerStarted","Data":"b418729532baafce988d27257cdd8b59e3636f156481082f20f7e0906e06d000"} Nov 01 23:25:17.816849 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:17.816739 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:17.837250 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:17.837199 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" podStartSLOduration=2.364560613 podStartE2EDuration="2.837185931s" totalImagesPullingTime="472.625318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:25:17.835792774 +0000 UTC m=+3683.263670705" watchObservedRunningTime="2026-11-01 23:25:17.837185931 +0000 UTC m=+3683.265063896" Nov 01 23:25:18.273219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:18.273174 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:19.072535 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:19.072438 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:19.806301 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:19.806265 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:20.471181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:20.471147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:21.053293 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:21.053256 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:21.697599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:21.697185 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:22.257097 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:22.257065 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:22.859357 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:22.859325 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:23.595612 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:23.595563 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:23.831904 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:23.831876 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:25:24.332891 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:24.332862 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:25.003447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:25.003418 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:25.835747 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:25.835699 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:26.380934 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:26.380888 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:27.064010 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.063967 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:27.757430 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.757401 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:27.873462 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.873432 4981 generic.go:356] "Generic (PLEG): container finished" podID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerID="cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344" exitCode=0 Nov 01 23:25:27.873670 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.873489 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" event={"ID":"4b35ffa6-b273-494c-b80f-7a4602de25f6","Type":"ContainerDied","Data":"cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344"} Nov 01 23:25:27.873936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.873894 4981 scope.go:122] "RemoveContainer" containerID="cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344" Nov 01 23:25:27.875122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.875092 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:25:27.875225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.875159 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:25:27.875225 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:27.875191 4981 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 23:25:28.554410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:28.554308 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:28.798599 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:28.797685 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:28.903969 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:28.903878 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" event={"ID":"4b35ffa6-b273-494c-b80f-7a4602de25f6","Type":"ContainerStarted","Data":"6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e"} Nov 01 23:25:29.199879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:29.199848 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:29.657207 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:29.657108 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:29.658706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:29.658671 4981 patch_prober.go:29] interesting pod/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Nov 01 23:25:29.658846 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:29.658728 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Nov 01 23:25:29.847919 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:29.847891 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:29.947991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:29.947960 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:30.570864 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:30.570830 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:30.657483 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:30.657443 4981 patch_prober.go:29] interesting pod/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Nov 01 23:25:30.658020 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:30.657506 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Nov 01 23:25:31.273437 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:31.273399 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:31.657767 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:31.657673 4981 patch_prober.go:29] interesting pod/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Nov 01 23:25:31.657767 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:31.657740 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Nov 01 23:25:32.199104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:32.199065 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:32.657837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:32.657748 4981 patch_prober.go:29] interesting pod/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Nov 01 23:25:32.657837 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:32.657805 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Nov 01 23:25:32.999111 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:32.999074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:33.657868 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:33.657835 4981 patch_prober.go:29] interesting pod/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Nov 01 23:25:33.658290 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:33.657890 4981 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Nov 01 23:25:33.726777 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:33.726736 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:34.589990 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:34.589956 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:34.661479 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:34.661453 4981 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:34.662025 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:34.661850 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:34.662468 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:34.662449 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:25:35.450304 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:35.450270 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:36.069644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:36.069610 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:36.775931 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:36.775895 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:37.461400 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:37.461369 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:38.139685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:38.139653 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:38.858278 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:38.858245 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:39.452944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:39.452911 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:40.062860 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:40.062826 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:40.761986 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:40.761952 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:41.365038 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:41.365004 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:42.009787 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:42.009737 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:42.765838 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:42.765801 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:43.649168 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:43.649135 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:44.269199 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:44.269139 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:44.523948 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:25:44.523854 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:25:44.950120 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:44.950085 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:45.540346 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:45.540310 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:46.191654 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:46.191623 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:46.935105 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:46.935074 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:47.565440 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:47.565405 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:48.240331 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:48.240299 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:48.862975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:48.862940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:49.550414 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:49.550379 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:50.233561 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:50.233532 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:50.850957 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:50.850924 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:51.653416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:51.653383 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:52.338180 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:52.338147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:52.958759 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:52.958725 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:53.626710 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:53.626678 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:54.166680 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:54.166648 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:54.865390 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:54.865359 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:55.475469 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:55.475435 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:56.058302 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:56.058268 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:56.769493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:56.769464 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:57.446088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:57.446054 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:58.100706 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:58.100664 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:58.783950 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:58.783921 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:25:59.554491 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:25:59.554462 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:00.167316 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:00.167283 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:00.814762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:00.814730 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:01.516520 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:01.516488 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:02.174145 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:02.174115 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:02.912975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:02.912938 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:03.524590 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:03.524536 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:04.050792 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:04.050759 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:04.567518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:04.567484 4981 scope.go:122] "RemoveContainer" containerID="3cbbdc31cd59e41c09048419ffaccb089e6a7ee321bde88b24e6019812dcf4c7" Nov 01 23:26:04.655432 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:04.655400 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:05.332143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:05.332108 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:05.950434 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:05.950391 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:06.556006 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:06.555969 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:07.165472 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:07.165438 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:07.867057 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:07.867027 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:08.456976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:08.456945 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:09.057153 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:09.057121 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:09.738736 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:09.738704 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:10.440820 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:10.440788 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:10.971075 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:10.971045 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:11.645512 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:11.645480 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:12.000309 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:12.000277 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:12.121547 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:12.121502 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:12.270991 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:12.270909 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:12.943103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:12.943070 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:13.559726 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:13.559690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:14.229848 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:14.229805 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:14.523767 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:26:14.523686 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:26:14.757863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:14.757833 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:15.365760 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:15.365732 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:15.970407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:15.970374 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:16.267674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:16.267549 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:16.570355 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:16.570275 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:17.257668 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:17.257637 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:17.867553 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:17.867522 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:18.550229 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:18.550196 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:19.223933 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:19.223901 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:19.850627 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:19.850592 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:20.464936 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:20.464904 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:21.019118 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:21.019077 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l"] Nov 01 23:26:21.023724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:21.023700 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwd5l"] Nov 01 23:26:21.137893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:21.137848 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:21.943042 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:21.943009 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:22.552143 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:22.552113 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:22.732672 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:22.732640 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a55ba9-05da-43bd-b577-27f7f346992b" path="/var/lib/kubelet/pods/c7a55ba9-05da-43bd-b577-27f7f346992b/volumes" Nov 01 23:26:23.164402 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:23.164349 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:23.882502 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:23.882440 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:24.590194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:24.590160 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:25.178382 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:25.178347 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:25.851416 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:25.851383 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:26.465060 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:26.465032 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:27.082852 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:27.082819 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:27.759954 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:27.759908 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:28.466703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:28.466624 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:29.079763 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:29.079728 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:29.763185 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:29.763152 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:30.658353 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:30.658304 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:31.255756 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:31.255720 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:31.943524 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:31.943474 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:32.610053 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:32.610018 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:33.159540 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:33.159504 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:33.830493 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:33.830464 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:34.464863 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:34.464827 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:35.133232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:35.133195 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:35.868409 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:35.868382 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:36.660635 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:36.660590 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:37.364410 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:37.364373 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:37.973437 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:37.973407 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:38.619076 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:38.619039 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:39.022015 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:39.021969 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx"] Nov 01 23:26:39.024066 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:39.024042 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jtwxx"] Nov 01 23:26:39.256866 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:39.256831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:39.863985 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:39.863940 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:40.550114 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:40.550079 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:40.732993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:40.732958 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a37203f-7ed3-45eb-b9e3-1f51dcdef54d" path="/var/lib/kubelet/pods/6a37203f-7ed3-45eb-b9e3-1f51dcdef54d/volumes" Nov 01 23:26:41.265062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:41.265030 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:41.875183 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:41.875147 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:42.465939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:42.465902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:43.146646 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:43.146614 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:43.764328 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:43.764296 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:44.467245 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:44.467208 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:44.526076 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:26:44.526033 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:26:45.151498 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:45.151467 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:45.831555 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:45.831521 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:46.458233 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:46.458204 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:47.067909 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:47.067877 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:47.754874 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:47.754835 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:48.358724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:48.358692 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:49.062396 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:49.062363 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:49.749676 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:49.749646 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:50.358814 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:50.358781 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:50.459755 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:50.459721 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:51.093089 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:51.093055 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:51.665893 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:51.665863 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:52.450294 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:52.450262 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:53.345938 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:53.345902 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:53.959656 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:53.959624 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:54.572633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:54.572600 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:55.345815 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:55.345774 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:55.958065 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:55.958034 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:56.643349 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:56.643313 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:57.257117 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:57.257086 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:57.951179 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:57.951146 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:58.547932 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:58.547897 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:59.166867 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:59.166831 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:26:59.855955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:26:59.855924 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:00.515625 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:00.515570 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:01.245817 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:01.245773 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:01.867116 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:01.867046 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:02.567725 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:02.567690 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:02.775993 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:02.775961 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:02.875834 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:02.875757 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:08.165972 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.165867 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj"] Nov 01 23:27:08.166545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.166129 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" podUID="2167f126-c1a8-4410-9622-52fd2b0ff06e" containerName="maas-api" containerID="cri-o://b418729532baafce988d27257cdd8b59e3636f156481082f20f7e0906e06d000" gracePeriod=30 Nov 01 23:27:08.211854 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.211825 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:08.344567 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.344538 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:08.413194 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.413168 4981 generic.go:356] "Generic (PLEG): container finished" podID="2167f126-c1a8-4410-9622-52fd2b0ff06e" containerID="b418729532baafce988d27257cdd8b59e3636f156481082f20f7e0906e06d000" exitCode=0 Nov 01 23:27:08.413330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.413235 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" event={"ID":"2167f126-c1a8-4410-9622-52fd2b0ff06e","Type":"ContainerDied","Data":"b418729532baafce988d27257cdd8b59e3636f156481082f20f7e0906e06d000"} Nov 01 23:27:08.413330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.413284 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" event={"ID":"2167f126-c1a8-4410-9622-52fd2b0ff06e","Type":"ContainerDied","Data":"ecfb29899d6fe6f42032954973e8ccca15d8cfa3cc1ea0c51827bb8b422b4b99"} Nov 01 23:27:08.413330 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.413299 4981 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecfb29899d6fe6f42032954973e8ccca15d8cfa3cc1ea0c51827bb8b422b4b99" Nov 01 23:27:08.419883 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.419832 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:27:08.458995 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.458923 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:08.480939 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.480902 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-maas-api-tls\") pod \"2167f126-c1a8-4410-9622-52fd2b0ff06e\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " Nov 01 23:27:08.481159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.480999 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2167f126-c1a8-4410-9622-52fd2b0ff06e-kube-api-access-4tkc4\" (UniqueName: \"kubernetes.io/projected/2167f126-c1a8-4410-9622-52fd2b0ff06e-kube-api-access-4tkc4\") pod \"2167f126-c1a8-4410-9622-52fd2b0ff06e\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " Nov 01 23:27:08.481159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.481082 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-metrics-certs\") pod \"2167f126-c1a8-4410-9622-52fd2b0ff06e\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " Nov 01 23:27:08.481159 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.481123 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2167f126-c1a8-4410-9622-52fd2b0ff06e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2167f126-c1a8-4410-9622-52fd2b0ff06e-openshift-service-ca\") pod \"2167f126-c1a8-4410-9622-52fd2b0ff06e\" (UID: \"2167f126-c1a8-4410-9622-52fd2b0ff06e\") " Nov 01 23:27:08.481914 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.481888 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2167f126-c1a8-4410-9622-52fd2b0ff06e-openshift-service-ca" pod "2167f126-c1a8-4410-9622-52fd2b0ff06e" (UID: "2167f126-c1a8-4410-9622-52fd2b0ff06e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:27:08.483975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.483948 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-metrics-certs" pod "2167f126-c1a8-4410-9622-52fd2b0ff06e" (UID: "2167f126-c1a8-4410-9622-52fd2b0ff06e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:27:08.483975 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.483970 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2167f126-c1a8-4410-9622-52fd2b0ff06e-kube-api-access-4tkc4" pod "2167f126-c1a8-4410-9622-52fd2b0ff06e" (UID: "2167f126-c1a8-4410-9622-52fd2b0ff06e"). InnerVolumeSpecName "kube-api-access-4tkc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:08.484246 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.484225 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-maas-api-tls" pod "2167f126-c1a8-4410-9622-52fd2b0ff06e" (UID: "2167f126-c1a8-4410-9622-52fd2b0ff06e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 23:27:08.582831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.582803 4981 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-metrics-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.582831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.582828 4981 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2167f126-c1a8-4410-9622-52fd2b0ff06e-openshift-service-ca\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.582831 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.582839 4981 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2167f126-c1a8-4410-9622-52fd2b0ff06e-maas-api-tls\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.583074 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.582848 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tkc4\" (UniqueName: \"kubernetes.io/projected/2167f126-c1a8-4410-9622-52fd2b0ff06e-kube-api-access-4tkc4\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:08.957392 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:08.957359 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:09.417702 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:09.417670 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj" Nov 01 23:27:09.434452 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:09.434417 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj"] Nov 01 23:27:09.440296 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:09.440271 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-ede84d-9c5bbff8d-d7hfj"] Nov 01 23:27:10.731896 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:10.731865 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2167f126-c1a8-4410-9622-52fd2b0ff06e" path="/var/lib/kubelet/pods/2167f126-c1a8-4410-9622-52fd2b0ff06e/volumes" Nov 01 23:27:14.522716 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:27:14.522675 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:27:14.601134 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:14.601104 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:14.701454 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:14.701421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:14.905518 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:14.905418 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg"] Nov 01 23:27:14.905724 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:14.905679 4981 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" containerID="cri-o://6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e" gracePeriod=30 Nov 01 23:27:14.937685 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:14.937657 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:16.316411 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:16.316382 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:19.664232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:19.664195 4981 patch_prober.go:29] interesting pod/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 23:27:19.664781 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:19.664260 4981 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 23:27:20.158563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.158541 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:27:20.161713 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161692 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-socket\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.161799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161762 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b35ffa6-b273-494c-b80f-7a4602de25f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b35ffa6-b273-494c-b80f-7a4602de25f6-istiod-ca-cert\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.161799 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161782 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-podinfo\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.161935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161818 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-kube-api-access-557ws\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-kube-api-access-557ws\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.161935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161866 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-credential-socket\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.161935 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161902 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-certs\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.162104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161953 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-token\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-token\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.162104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161982 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-socket" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:27:20.162104 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.161993 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-data\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.162268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162137 4981 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-envoy\") pod \"4b35ffa6-b273-494c-b80f-7a4602de25f6\" (UID: \"4b35ffa6-b273-494c-b80f-7a4602de25f6\") " Nov 01 23:27:20.162268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162213 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-data" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:27:20.162268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162228 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b35ffa6-b273-494c-b80f-7a4602de25f6-istiod-ca-cert" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 23:27:20.162268 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162230 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-credential-socket" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:27:20.162494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162284 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-certs" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:27:20.162494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162467 4981 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-credential-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.162494 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162485 4981 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-certs\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.162679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162499 4981 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-data\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.162679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162510 4981 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-workload-socket\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.162679 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.162518 4981 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b35ffa6-b273-494c-b80f-7a4602de25f6-istiod-ca-cert\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.164388 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.164363 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-podinfo" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 23:27:20.164522 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.164395 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-kube-api-access-557ws" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "kube-api-access-557ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:20.164600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.164555 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-envoy" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 23:27:20.165088 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.165069 4981 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-token" pod "4b35ffa6-b273-494c-b80f-7a4602de25f6" (UID: "4b35ffa6-b273-494c-b80f-7a4602de25f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 23:27:20.263869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.263838 4981 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-envoy\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.263869 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.263868 4981 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-podinfo\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.264083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.263883 4981 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-557ws\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-kube-api-access-557ws\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.264083 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.263896 4981 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b35ffa6-b273-494c-b80f-7a4602de25f6-istio-token\") on node \"ip-10-0-138-87.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 23:27:20.478241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.478151 4981 generic.go:356] "Generic (PLEG): container finished" podID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerID="6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e" exitCode=0 Nov 01 23:27:20.478241 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.478203 4981 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" Nov 01 23:27:20.478451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.478229 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" event={"ID":"4b35ffa6-b273-494c-b80f-7a4602de25f6","Type":"ContainerDied","Data":"6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e"} Nov 01 23:27:20.478451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.478309 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg" event={"ID":"4b35ffa6-b273-494c-b80f-7a4602de25f6","Type":"ContainerDied","Data":"8a875a16c92a951b89056bf0deccfaf7afb9b5787e39c698d987d8f5aa343469"} Nov 01 23:27:20.478451 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.478333 4981 scope.go:122] "RemoveContainer" containerID="6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e" Nov 01 23:27:20.488407 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.488389 4981 scope.go:122] "RemoveContainer" containerID="cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344" Nov 01 23:27:20.497136 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.497118 4981 scope.go:122] "RemoveContainer" containerID="6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e" Nov 01 23:27:20.497398 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:27:20.497378 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e\": container with ID starting with 6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e not found: ID does not exist" containerID="6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e" Nov 01 23:27:20.497447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.497408 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e"} err="failed to get container status \"6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e\": rpc error: code = NotFound desc = could not find container \"6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e\": container with ID starting with 6682214f7c4c86d47a9b63411baf562d2997b26f9df9c8cf4b2ed642d3c0df6e not found: ID does not exist" Nov 01 23:27:20.497447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.497426 4981 scope.go:122] "RemoveContainer" containerID="cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344" Nov 01 23:27:20.497682 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:27:20.497664 4981 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344\": container with ID starting with cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344 not found: ID does not exist" containerID="cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344" Nov 01 23:27:20.497735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.497688 4981 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344"} err="failed to get container status \"cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344\": rpc error: code = NotFound desc = could not find container \"cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344\": container with ID starting with cd9cd767ffc76fbce36f50bf6cdf39f7e7bdf7aa3265dda47c6031ad7164e344 not found: ID does not exist" Nov 01 23:27:20.500825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.500805 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg"] Nov 01 23:27:20.504122 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.504101 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-ede84d-openshift-default-694897fbcc-jmtlg"] Nov 01 23:27:20.732927 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:20.732852 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" path="/var/lib/kubelet/pods/4b35ffa6-b273-494c-b80f-7a4602de25f6/volumes" Nov 01 23:27:32.905317 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:32.905281 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:38.068092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:38.068055 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:27:44.525098 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:27:44.522515 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:27:51.502187 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:27:51.502138 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:28:04.628600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:04.628248 4981 scope.go:122] "RemoveContainer" containerID="6cbf8a0638b1fb09e069dad753419fd2b1805ab5bdf1e8650f2a2fcd35eec1f4" Nov 01 23:28:04.652538 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:04.652513 4981 scope.go:122] "RemoveContainer" containerID="eabc21de3c7c2d676d0aa6dc5fed6a41ae3f685bfd63da780681b6b91edcf565" Nov 01 23:28:14.006045 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:14.006009 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767cb84c4f-zld2b_bc8e43f0-c141-4ba7-a4d1-cdc273722c58/manager/0.log" Nov 01 23:28:14.524866 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:28:14.524829 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:28:15.021486 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:15.021461 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cxr2r_1406f61b-2ddc-402f-8631-3b00159964b0/server/0.log" Nov 01 23:28:15.199435 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:15.199406 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8wbsw_80467d4e-cd6c-479b-8334-daeb938706b1/manager/0.log" Nov 01 23:28:16.084636 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:16.084602 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7587df7975-7qwpp_27322fe7-25cf-4c7a-8733-8a0150346e19/maas-api/0.log" Nov 01 23:28:16.334621 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:16.334585 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:28:16.593078 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:16.593047 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-t6xjq_86452723-09a0-4e7e-8f6c-21c2c93a3439/revoke-keys/0.log" Nov 01 23:28:16.762435 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:16.762405 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-6zgsv_82f93db3-fe39-413f-8fa3-3c1746108d62/revoke-keys/0.log" Nov 01 23:28:17.099697 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:17.099668 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-97012f4d-k8gz2_0869d024-1d26-415b-b8a8-4d71ce3dee2d/revoke-keys/0.log" Nov 01 23:28:19.339855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:19.339825 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mqgpf_b82fae95-f18b-4064-a70b-3cf1a89efc73/kuadrant-console-plugin/0.log" Nov 01 23:28:21.109873 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:21.109842 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-pjzcq_309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a/kube-auth-proxy/0.log" Nov 01 23:28:21.283545 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:21.283517 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6tll8_3684469b-976a-4854-9697-a4d97edc5b16/istio-proxy/0.log" Nov 01 23:28:21.437703 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:21.437674 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-7v6z5_4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7/istio-proxy/0.log" Nov 01 23:28:22.268614 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:22.268563 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lt55h_68b388b0-9081-4e9f-9bc4-c0f4b91397fd/istio-proxy/0.log" Nov 01 23:28:23.155469 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:23.155437 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sph7r_d3deba2b-ac37-495d-bee9-e015774a9626/payload-pre-processing/0.log" Nov 01 23:28:23.320979 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:23.320951 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:28:24.191030 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:24.190996 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn_7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23/main/0.log" Nov 01 23:28:24.864840 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:24.864812 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m_c6f43edf-54c6-41d3-8445-a8d2aa81a230/main/0.log" Nov 01 23:28:25.332968 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:25.332928 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8_e7f7e966-fddb-4b4c-b76e-f2c2f55c8350/main/0.log" Nov 01 23:28:34.700731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:34.700703 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767cb84c4f-zld2b_bc8e43f0-c141-4ba7-a4d1-cdc273722c58/manager/0.log" Nov 01 23:28:35.634570 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:35.634538 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cxr2r_1406f61b-2ddc-402f-8631-3b00159964b0/server/0.log" Nov 01 23:28:35.810851 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:35.810816 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-8wbsw_80467d4e-cd6c-479b-8334-daeb938706b1/manager/0.log" Nov 01 23:28:36.476956 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:36.476927 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7587df7975-7qwpp_27322fe7-25cf-4c7a-8733-8a0150346e19/maas-api/0.log" Nov 01 23:28:37.003606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:37.003556 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-t6xjq_86452723-09a0-4e7e-8f6c-21c2c93a3439/revoke-keys/0.log" Nov 01 23:28:37.167259 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:37.167229 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-6zgsv_82f93db3-fe39-413f-8fa3-3c1746108d62/revoke-keys/0.log" Nov 01 23:28:37.509618 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:37.509560 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-97012f4d-k8gz2_0869d024-1d26-415b-b8a8-4d71ce3dee2d/revoke-keys/0.log" Nov 01 23:28:39.409633 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:39.409595 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mqgpf_b82fae95-f18b-4064-a70b-3cf1a89efc73/kuadrant-console-plugin/0.log" Nov 01 23:28:40.967211 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:40.967180 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-pjzcq_309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a/kube-auth-proxy/0.log" Nov 01 23:28:41.134892 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:41.134861 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6tll8_3684469b-976a-4854-9697-a4d97edc5b16/istio-proxy/0.log" Nov 01 23:28:41.310931 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:41.310856 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-7v6z5_4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7/istio-proxy/0.log" Nov 01 23:28:42.161009 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:42.160979 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lt55h_68b388b0-9081-4e9f-9bc4-c0f4b91397fd/istio-proxy/0.log" Nov 01 23:28:42.972037 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:42.972012 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sph7r_d3deba2b-ac37-495d-bee9-e015774a9626/payload-pre-processing/0.log" Nov 01 23:28:43.145062 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:43.145032 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:28:43.815350 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:43.815317 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-5xksn_7109922b-7a0a-4ba7-b7a6-a49dcf5e6a23/main/0.log" Nov 01 23:28:44.460816 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:44.460784 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ncp7m_c6f43edf-54c6-41d3-8445-a8d2aa81a230/main/0.log" Nov 01 23:28:44.523783 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:28:44.523747 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:28:44.968313 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:44.968283 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-8k9v8_e7f7e966-fddb-4b4c-b76e-f2c2f55c8350/main/0.log" Nov 01 23:28:48.019176 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:48.019140 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv"] Nov 01 23:28:48.022012 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:48.021955 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6zgsv"] Nov 01 23:28:48.732563 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:28:48.732530 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f93db3-fe39-413f-8fa3-3c1746108d62" path="/var/lib/kubelet/pods/82f93db3-fe39-413f-8fa3-3c1746108d62/volumes" Nov 01 23:29:04.021775 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:04.021737 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq"] Nov 01 23:29:04.027825 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:04.027792 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-t6xjq"] Nov 01 23:29:04.734730 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:04.734253 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86452723-09a0-4e7e-8f6c-21c2c93a3439" path="/var/lib/kubelet/pods/86452723-09a0-4e7e-8f6c-21c2c93a3439/volumes" Nov 01 23:29:11.457952 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:11.457909 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:29:11.570890 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:11.570854 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:29:14.522495 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:29:14.522460 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:29:16.395394 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:16.395363 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:29:21.284630 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:21.284600 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:29:21.446620 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:21.446594 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:29:24.021080 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:24.021042 4981 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2"] Nov 01 23:29:24.024275 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:24.024247 4981 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97012f4d-k8gz2"] Nov 01 23:29:24.733847 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:29:24.733807 4981 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0869d024-1d26-415b-b8a8-4d71ce3dee2d" path="/var/lib/kubelet/pods/0869d024-1d26-415b-b8a8-4d71ce3dee2d/volumes" Nov 01 23:29:44.522808 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:29:44.522774 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:30:04.723977 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:04.723942 4981 scope.go:122] "RemoveContainer" containerID="751bcbfc5ad81464ec03bd9a8d3ba5e8a455245f9362958f5c525e7029f6e96d" Nov 01 23:30:04.733166 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:04.733148 4981 scope.go:122] "RemoveContainer" containerID="592b4a6dfafb35ebd72094818c28b6c8ee9ea59535ab4bddbe1c7ee6ac134b73" Nov 01 23:30:06.718811 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:06.718782 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:30:06.893016 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:06.892991 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:30:07.155898 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.155816 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7bxbb/must-gather-bjfrc"] Nov 01 23:30:07.156272 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156259 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" Nov 01 23:30:07.156318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156273 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" Nov 01 23:30:07.156318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156296 4981 cpu_manager.go:414] "containerMap: removing container" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" Nov 01 23:30:07.156318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156301 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" Nov 01 23:30:07.156318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156312 4981 cpu_manager.go:414] "containerMap: removing container" podUID="2167f126-c1a8-4410-9622-52fd2b0ff06e" containerName="maas-api" Nov 01 23:30:07.156318 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156317 4981 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2167f126-c1a8-4410-9622-52fd2b0ff06e" containerName="maas-api" Nov 01 23:30:07.156515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156376 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="2167f126-c1a8-4410-9622-52fd2b0ff06e" containerName="maas-api" Nov 01 23:30:07.156515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156387 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" Nov 01 23:30:07.156515 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.156397 4981 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b35ffa6-b273-494c-b80f-7a4602de25f6" containerName="istio-proxy" Nov 01 23:30:07.161223 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.161200 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.166780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.166649 4981 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7bxbb\"/\"default-dockercfg-k9lzk\"" Nov 01 23:30:07.166780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.166730 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bxbb\"/\"kube-root-ca.crt\"" Nov 01 23:30:07.166780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.166758 4981 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7bxbb\"/\"openshift-service-ca.crt\"" Nov 01 23:30:07.168808 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.168780 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/must-gather-bjfrc"] Nov 01 23:30:07.185446 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.185421 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:30:07.246976 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.246942 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd-must-gather-output\") pod \"must-gather-bjfrc\" (UID: \"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd\") " pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.247182 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.247056 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226c2\" (UniqueName: \"kubernetes.io/projected/8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd-kube-api-access-226c2\") pod \"must-gather-bjfrc\" (UID: \"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd\") " pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.284181 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.284151 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:30:07.348325 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.348285 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd-must-gather-output\") pod \"must-gather-bjfrc\" (UID: \"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd\") " pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.348523 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.348360 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226c2\" (UniqueName: \"kubernetes.io/projected/8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd-kube-api-access-226c2\") pod \"must-gather-bjfrc\" (UID: \"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd\") " pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.348694 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.348672 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd-must-gather-output\") pod \"must-gather-bjfrc\" (UID: \"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd\") " pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.357092 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.357067 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226c2\" (UniqueName: \"kubernetes.io/projected/8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd-kube-api-access-226c2\") pod \"must-gather-bjfrc\" (UID: \"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd\") " pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.479600 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.479538 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/must-gather-bjfrc" Nov 01 23:30:07.760902 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.760821 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/must-gather-bjfrc"] Nov 01 23:30:07.763768 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:30:07.763738 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e8c9c68_50b9_4165_a2bf_aeee7abcb4dd.slice/crio-f7cf7254a623173d875de809d9ec1a338f750e69fd4f16174a3f039d882a64b7 WatchSource:0}: Error finding container f7cf7254a623173d875de809d9ec1a338f750e69fd4f16174a3f039d882a64b7: Status 404 returned error can't find the container with id f7cf7254a623173d875de809d9ec1a338f750e69fd4f16174a3f039d882a64b7 Nov 01 23:30:07.765569 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:07.765552 4981 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 23:30:08.365644 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:08.365611 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/must-gather-bjfrc" event={"ID":"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd","Type":"ContainerStarted","Data":"f7cf7254a623173d875de809d9ec1a338f750e69fd4f16174a3f039d882a64b7"} Nov 01 23:30:09.375139 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:09.375096 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/must-gather-bjfrc" event={"ID":"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd","Type":"ContainerStarted","Data":"d5bdeeb9f4a88ca0d685a057eb90a5b8a8e95ee9fb65b007cfa078c1e8595f78"} Nov 01 23:30:09.375622 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:09.375156 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/must-gather-bjfrc" event={"ID":"8e8c9c68-50b9-4165-a2bf-aeee7abcb4dd","Type":"ContainerStarted","Data":"60084f03eeea711800ba5573ecda92b607236ce4fe4aed1e9f8ac727f94c01fe"} Nov 01 23:30:10.610311 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:10.610280 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wdkm9_7caa0db8-4f8f-4848-a2a6-845f20c9a888/global-pull-secret-syncer/0.log" Nov 01 23:30:10.745405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:10.745364 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" Nov 01 23:30:10.856043 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:10.854248 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-87.us-east-2.compute.internal_9bc77fd8b25c4cfddf6fa5b50fb587e4/haproxy/0.log" Nov 01 23:30:14.606959 ip-10-0-138-87 kubenswrapper[4981]: E1101 23:30:14.606919 4981 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m9kr7" containerName="extractor" Nov 01 23:30:16.447594 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:16.446810 4981 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rvph_7025fde7-1830-4d2e-9780-a4c0a8195d0f/konnectivity-agent/0.log" limit=1024 Nov 01 23:30:18.057203 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:18.057172 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mqgpf_b82fae95-f18b-4064-a70b-3cf1a89efc73/kuadrant-console-plugin/0.log" Nov 01 23:30:19.868606 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:19.866009 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bvp46_2e4ab4e4-564d-4403-bc0e-802ba2a2d29f/kube-state-metrics/0.log" Nov 01 23:30:19.888882 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:19.888848 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bvp46_2e4ab4e4-564d-4403-bc0e-802ba2a2d29f/kube-rbac-proxy-main/0.log" Nov 01 23:30:19.914368 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:19.914338 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bvp46_2e4ab4e4-564d-4403-bc0e-802ba2a2d29f/kube-rbac-proxy-self/0.log" Nov 01 23:30:20.097743 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.097712 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhtnf_e0d574c8-9d38-43ba-99ea-c24167bf452b/node-exporter/0.log" Nov 01 23:30:20.127389 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.127308 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhtnf_e0d574c8-9d38-43ba-99ea-c24167bf452b/kube-rbac-proxy/0.log" Nov 01 23:30:20.154659 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.154623 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhtnf_e0d574c8-9d38-43ba-99ea-c24167bf452b/init-textfile/0.log" Nov 01 23:30:20.269678 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.269649 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m2b87_31cc6987-5648-4628-a50a-6cf2f1606627/kube-rbac-proxy-main/0.log" Nov 01 23:30:20.297391 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.297366 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m2b87_31cc6987-5648-4628-a50a-6cf2f1606627/kube-rbac-proxy-self/0.log" Nov 01 23:30:20.323529 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.323461 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m2b87_31cc6987-5648-4628-a50a-6cf2f1606627/openshift-state-metrics/0.log" Nov 01 23:30:20.366217 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.366185 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49fa04e-24ec-496c-958b-c268bf039b7e/prometheus/0.log" Nov 01 23:30:20.390497 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.390423 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49fa04e-24ec-496c-958b-c268bf039b7e/config-reloader/0.log" Nov 01 23:30:20.414944 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.414902 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49fa04e-24ec-496c-958b-c268bf039b7e/thanos-sidecar/0.log" Nov 01 23:30:20.446565 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.446537 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49fa04e-24ec-496c-958b-c268bf039b7e/kube-rbac-proxy-web/0.log" Nov 01 23:30:20.475035 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.475001 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49fa04e-24ec-496c-958b-c268bf039b7e/kube-rbac-proxy/0.log" Nov 01 23:30:20.497796 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.497768 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49fa04e-24ec-496c-958b-c268bf039b7e/kube-rbac-proxy-thanos/0.log" Nov 01 23:30:20.521167 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.521144 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e49fa04e-24ec-496c-958b-c268bf039b7e/init-config-reloader/0.log" Nov 01 23:30:20.601519 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.601473 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2k4xp_5c512618-f7bd-4210-90f9-e525e4a8a1b7/prometheus-operator-admission-webhook/0.log" Nov 01 23:30:20.636905 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.636873 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69d449f8cd-dgrw9_c95c3ce6-d144-4423-94fc-65f47402f703/telemeter-client/0.log" Nov 01 23:30:20.667273 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.667167 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7bxbb/must-gather-bjfrc" podStartSLOduration=12.73769508 podStartE2EDuration="13.667146523s" totalImagesPullingTime="929.451443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:09.393264512 +0000 UTC m=+3974.821142443" watchObservedRunningTime="2026-11-01 23:30:20.667146523 +0000 UTC m=+3986.095024456" Nov 01 23:30:20.668378 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.668343 4981 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf"] Nov 01 23:30:20.671735 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.671706 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69d449f8cd-dgrw9_c95c3ce6-d144-4423-94fc-65f47402f703/reload/0.log" Nov 01 23:30:20.676855 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.676823 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.685154 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.685129 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf"] Nov 01 23:30:20.709219 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.709191 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69d449f8cd-dgrw9_c95c3ce6-d144-4423-94fc-65f47402f703/kube-rbac-proxy/0.log" Nov 01 23:30:20.743055 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.743028 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887556c4f-ppfhf_3e7900c2-b948-46db-a1fa-535cf450e73b/thanos-query/0.log" Nov 01 23:30:20.790267 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.790236 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887556c4f-ppfhf_3e7900c2-b948-46db-a1fa-535cf450e73b/kube-rbac-proxy-web/0.log" Nov 01 23:30:20.815381 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.815348 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887556c4f-ppfhf_3e7900c2-b948-46db-a1fa-535cf450e73b/kube-rbac-proxy/0.log" Nov 01 23:30:20.822398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.822367 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-lib-modules\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.822561 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.822413 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-sys\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.822561 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.822534 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kc9j\" (UniqueName: \"kubernetes.io/projected/a90fd844-fb35-4afd-8eae-e989b844f3af-kube-api-access-2kc9j\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.822671 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.822649 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-proc\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.822731 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.822717 4981 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-podres\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.841762 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.841733 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887556c4f-ppfhf_3e7900c2-b948-46db-a1fa-535cf450e73b/prom-label-proxy/0.log" Nov 01 23:30:20.867674 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.867646 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887556c4f-ppfhf_3e7900c2-b948-46db-a1fa-535cf450e73b/kube-rbac-proxy-rules/0.log" Nov 01 23:30:20.893132 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.893102 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6887556c4f-ppfhf_3e7900c2-b948-46db-a1fa-535cf450e73b/kube-rbac-proxy-metrics/0.log" Nov 01 23:30:20.924566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.923437 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kc9j\" (UniqueName: \"kubernetes.io/projected/a90fd844-fb35-4afd-8eae-e989b844f3af-kube-api-access-2kc9j\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.924566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.923544 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-proc\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.924566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.923675 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-podres\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.924566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.923723 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-lib-modules\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.924566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.923753 4981 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-sys\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.924566 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.923866 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-sys\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.925028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.924633 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-podres\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.925028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.924704 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-proc\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.925028 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.924771 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a90fd844-fb35-4afd-8eae-e989b844f3af-lib-modules\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:20.935447 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:20.935420 4981 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kc9j\" (UniqueName: \"kubernetes.io/projected/a90fd844-fb35-4afd-8eae-e989b844f3af-kube-api-access-2kc9j\") pod \"perf-node-gather-daemonset-27xvf\" (UID: \"a90fd844-fb35-4afd-8eae-e989b844f3af\") " pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:21.005804 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:21.005764 4981 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:21.318780 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:21.317434 4981 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf"] Nov 01 23:30:21.322056 ip-10-0-138-87 kubenswrapper[4981]: W1101 23:30:21.322011 4981 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda90fd844_fb35_4afd_8eae_e989b844f3af.slice/crio-651b3e0fe22c16715aa89ee7cbf472f417ecd84b2af1da5e064ac6902ee274c9 WatchSource:0}: Error finding container 651b3e0fe22c16715aa89ee7cbf472f417ecd84b2af1da5e064ac6902ee274c9: Status 404 returned error can't find the container with id 651b3e0fe22c16715aa89ee7cbf472f417ecd84b2af1da5e064ac6902ee274c9 Nov 01 23:30:21.488385 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:21.488336 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" event={"ID":"a90fd844-fb35-4afd-8eae-e989b844f3af","Type":"ContainerStarted","Data":"51036f08759ad493cd0a222a7ab289671b375c848564c28ec078640c1a7a5afd"} Nov 01 23:30:21.488609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:21.488403 4981 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" event={"ID":"a90fd844-fb35-4afd-8eae-e989b844f3af","Type":"ContainerStarted","Data":"651b3e0fe22c16715aa89ee7cbf472f417ecd84b2af1da5e064ac6902ee274c9"} Nov 01 23:30:21.488609 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:21.488434 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:21.507511 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:21.506406 4981 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" podStartSLOduration=1.506386834 podStartE2EDuration="1.506386834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:30:21.505701487 +0000 UTC m=+3986.933579419" watchObservedRunningTime="2026-11-01 23:30:21.506386834 +0000 UTC m=+3986.934264767" Nov 01 23:30:23.258415 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:23.258386 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-n87cr_8deca68d-1a0b-4841-9c91-0682262d59c7/download-server/0.log" Nov 01 23:30:25.013376 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:25.013341 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5v8l4_874207d9-9be6-4544-bc12-2bf935d7e477/dns/0.log" Nov 01 23:30:25.036698 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:25.034944 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5v8l4_874207d9-9be6-4544-bc12-2bf935d7e477/kube-rbac-proxy/0.log" Nov 01 23:30:25.170413 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:25.170375 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n8wtc_948069e6-a5b8-407d-a12b-198a76a2e783/dns-node-resolver/0.log" Nov 01 23:30:25.851232 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:25.851198 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d7bc7f98b-p87lt_83ec6a1b-db92-4103-adec-8539c8bb86c2/registry/0.log" Nov 01 23:30:25.907342 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:25.907317 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnr7k_f130f8c5-0161-4ef2-8634-cc150d09ddf0/node-ca/0.log" Nov 01 23:30:27.506955 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:27.506922 4981 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7bxbb/perf-node-gather-daemonset-27xvf" Nov 01 23:30:29.214405 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:29.214376 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-pjzcq_309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a/kube-auth-proxy/0.log" Nov 01 23:30:29.253537 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:29.253502 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6tll8_3684469b-976a-4854-9697-a4d97edc5b16/istio-proxy/0.log" Nov 01 23:30:29.282370 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:29.282347 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-7v6z5_4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7/istio-proxy/0.log" Nov 01 23:30:29.450953 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:29.450922 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lt55h_68b388b0-9081-4e9f-9bc4-c0f4b91397fd/istio-proxy/0.log" Nov 01 23:30:29.614662 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:29.614564 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sph7r_d3deba2b-ac37-495d-bee9-e015774a9626/payload-pre-processing/0.log" Nov 01 23:30:29.643282 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:29.643251 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:30:32.374103 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:32.374071 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-82z6g_4e15bc3a-dc26-4b68-bba9-b670d741c952/serve-healthcheck-canary/0.log" Nov 01 23:30:32.454807 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:32.454777 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bc8df847-pjzcq_309eb3d8-3c5b-4ee7-bc7e-44b6ab6d7a8a/kube-auth-proxy/0.log" Nov 01 23:30:32.476823 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:32.476793 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6tll8_3684469b-976a-4854-9697-a4d97edc5b16/istio-proxy/0.log" Nov 01 23:30:32.493769 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:32.493740 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-7v6z5_4d2cb3a1-3ea5-436c-b204-c3f337bb2ec7/istio-proxy/0.log" Nov 01 23:30:32.618821 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:32.618784 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lt55h_68b388b0-9081-4e9f-9bc4-c0f4b91397fd/istio-proxy/0.log" Nov 01 23:30:32.735978 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:32.735904 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-sph7r_d3deba2b-ac37-495d-bee9-e015774a9626/payload-pre-processing/0.log" Nov 01 23:30:32.758568 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:32.758529 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-gzxqz_fd1ade81-64b5-4dcf-a5da-55632908568b/payload-processing/0.log" Nov 01 23:30:33.080240 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:33.080169 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9kr7_d9cf0fb4-ef5d-4921-9c01-8830c386dce8/kube-rbac-proxy/0.log" Nov 01 23:30:33.102398 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:33.102373 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9kr7_d9cf0fb4-ef5d-4921-9c01-8830c386dce8/exporter/0.log" Nov 01 23:30:33.127554 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:33.127528 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9kr7_d9cf0fb4-ef5d-4921-9c01-8830c386dce8/extractor/0.log" Nov 01 23:30:35.524981 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:35.524954 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5b59bbd84f-k42k8_2af45373-36c6-4d05-93c1-1a0a1805e7c2/manager/0.log" Nov 01 23:30:35.546049 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:35.546021 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5b59bbd84f-k42k8_2af45373-36c6-4d05-93c1-1a0a1805e7c2/copy-manifests/0.log" Nov 01 23:30:36.887879 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:36.887847 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-656887d9d4-psjhj_b3464466-11fd-4690-bd6a-7d6a80854fac/manager/0.log" Nov 01 23:30:36.918231 ip-10-0-138-87 kubenswrapper[4981]: I1101 23:30:36.918207 4981 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-7l5cw_74dc6f3c-838b-4ea7-9b81-3f0890e8955b/jobset-operator/0.log"